TechOpsGuys.com Diggin' technology every day

13Jan/14Off

BigCo Security: Fighting a war you cannot win

TechOps Guy: Nate

It has been somewhat interesting to watch how security vulnerabilities have evolved over the past twenty years or so that I've been working with computers anyway. For the most part in the early days security exploits were pretty harmless. Maybe your company got hacked to leverage it's bandwidth/disk space for pirated software or something like that.

The past several years though the rise in organized cyber crime and highly sophisticated attacks (even attacks from folks that some may consider friendly) is rather alarming. I do feel sorry for those in the security field, especially those at bigger organizations, whom by nature are bigger targets. They are (for the most part) fighting a war they simply cannot win.  Sooner or later they will be breached, and one interesting stat I heard last year at a presentation given by the CTO of Trend Micro was that the average attacker has access to a network to 210 days before being detected.

Companies can spend millions to billions of dollars on equipment, training, and staffing to protect themselves but it'll never be enough. I mean look no further than the NSA and Snowden? How much did he get away with again? The NSA admits they don't even know.

I wish the company that sponsored the event had published a video of this CTO presentation as I thought it was the most interesting I had/seen heard in years.  Here is another video from another event that he presented at, also quite good - though not as long as the presentation I saw.

Some details on a highly sophisticated successful attack executed against Korean banks targeting multiple platforms

Some details on a highly sophisticated successful attack executed against Korean banks targeting multiple platforms

The slide above shows a very large scale attack which had more than seventy custom malware packages built for it!

The recent highly sophisticated attacks against Target and Neiman Marcus are of course just the recent high profile examples.

The security of SCADA systems has long been a problem as well.

Over 60,000 exposed control systems found online.

Researchers have found vulnerabilities in industrial control systems that they say grant full control of systems running energy, chemical and transportation systems.

Speaking of industrial control systems, going back to the Trend Micro presentation they mentioned how they purchased some similar equipment to do some testing with. Their first tests involved a water pressure control station connected to the internet and they just watched to see who tried to attack it. This was a real system (not connected to any water source or supporting anybody).

Trend Micro tests who attacks their water pressure control system

Trend Micro tests who attacks their water pressure control system

One of the interesting bits was he noted that although there were a large number of attacks from China most of them were simply probing for information, they were not destructive. I don't remember who had the destructive attacks I want to say Laos and the U.S. but I could be wrong. He said since this test was so successful they were planning (perhaps already had) to purchase several more of these and place them around the world for monitoring.

I've never been too deep in security, I can count on one hand the number of times I've had to deal with a compromised system over the past 15 years(most recent one was a couple of months ago). Taking real basic security precautions protects you against a very large number of threats by default(with the most recent attack I dealt with I noted at least three best practices any of which would of prevented the attack from occurring, all of which would of had no impact to the system or application though none were in place), though at the end of the day your best defense against a targeted attack - is don't be a target to begin with. Obviously that is impossible for big organizations.

The recent DDoS attacks against gaming companies I believe impacted the company I work for, not because we are a gaming company but because we share the same ISP. The ISP responded quite well to the attacks in my opinion and later wrote a letter to all customers describing the attacks - an NTP amplification attack that exceeded 100Gbps in volume, the largest attack they had ever seen. It's the first DOS attack that has impacted stuff I operate that I've ever experienced to my knowledge.

Tagged as: Comments Off
21May/13Off

SHOCKER! Power grid vulernable to Cyberattack!

TechOps Guy: Nate

Yeah, it shouldn't be news.. but I guess I am sort of glad it is making some sort of headline. I have written in the past how I think the concept of a smart grid is terrible due to security concerns. I just have no confidence in today's security technology to properly secure such a system. If we can't properly secure our bank transactions(my main credit card was compromised for at least the 2nd or 3rd time this year and I am careful), how can anyone expect to secure the grid?

Just saw a new post on Slashdot which points to a new report being released that covers some of how vulnerable we are to attack on our grid.

The report, released ahead of a House hearing on cybersecurity by Congressmen Edward Markey (D-Mass.) and Henry Waxman (D-Calif.), finds that cyberattacks are a daily occurrence, with one power company claiming it fights off 10,000 attempted intrusions each month.

[..]

Such attacks could cut power to large sections of the country and take months to repair.

Oh how I miss the days of early cyber security where the threat was little more than kids poking around and learning. These days there is really little defense against the organized military of the likes of China, sigh.

If they want to get you, most likely they are going to get you.

I've had a discussion or two with a friend who works with industrial control systems and the security on those is generally worse than I had heard about with the various breaches around the world.

I don't see any real value the so called smart grid has, anything remotely resembling gains that would offset the massive growth of the network access points that are connected to the grid.

It's probably already too late. All security is some form of obscurity at the end of the day whether it is a password, or encryption or physical isolation. Obscuring the grid by reducing the network connections to it has got to provide some level of benefit...

5May/11Off

Sony Compromised by Apache bug?

TechOps Guy: Nate

Came across an article from a friend that talks about how Sony thinks they were compromised.

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which "was unpatched and had no firewall installed."

The firewall part is what gets me. Assuming of course this web server(s) were meant to be public, no firewall is going to protect you against this sort of thing since of course firewalls protecting public web servers have holes opened explicitly for the web server so all traffic is passed right through.

And I highly doubt those Apache web servers had confidential data as the article implies, obviously that data was on back end systems running databases of some sort.

Then there are people out there spouting stuff on PCI saying the automated external scans should of detected they were running outdated versions of software. In my experience such scans are really not worth much with Linux, primarily because they have no way to take into account patches that are back ported to the operating system. I've had a few arguments with security scanners trying to explain how a system is patched because the fix was back ported but them not being able to comprehend that because the major/minor version being reported by the server has not changed.

Then there was the company I worked for who had a web app that returned a HTTP/200 for pretty much everything, including things like 404s. This tripped every single alarm the scanners had, and they went nuts. And once again we had to explain that those windows exploits aren't going to work against our Apache Tomcat systems running Linux.

IDS and IPS are overrated as well, unless you really have the staff to watch and manage it full time. In all of the years I have worked at companies that deployed some sort of IDS (never IPS), I have seen it work, one time, back in I want to say 2002, I saw a dramatic upsurge in some type of traffic on our Snort IDS at the time from one particular host and turns out it had a virus on it. I worked at one company that was compromised at LEAST twice while I was there(on systems that weren't being properly managed). and of course the IDS never detected a thing. Then that company deployed(after I left) a higher end hardware-based IPS, and when they put it inline to the network (in passive, not enforcing mode) for some reason the IPS started dropping all SSL traffic for no reason.

They aren't completely useless though, they can help detect and sometimes protect against the more obvious types of attacks (SQL injection etc).  But in the grand scheme of things, especially when dealing with customized applications (not off the shelf like Exchange, Oracle or whatever), IDS/IPS and even firewalls provide only a tiny layer of additional security on top of good application design, good deployment practices(e.g. don't run as root, disable or remove subsystems that are not used, such as the management app in Tomcat, use encryption where possible), and a good authentication system for system level access (e.g. ssh keys). With regards to web applications, a good load balancer is more than adequate to protect the vast majority of applications out there, it is "firewall like" as in it only passes certain ports to the back end systems, but (for higher traffic sites this is important) vastly outperforms firewalls, which can be a massive bottleneck for front end systems.

With regards to the company that was compromised at least twice, the intrusion was minor and limited to a single system, the compromise occurred because the engineer who installed the system put it outside of the load balancers, it was a FTP server, or was it a monitoring server, I forgot.  Because it needed to be accessed externally the engineer thought hey let's just put it on the internet. Well it sat there for a good year or two, (never being patched in the meantime) before I joined the company, compromised in some fashion, and ssh was replaced with a trojaned copy (it was pretty obvious, I am assuming it was some sort of worm exploiting ssh). It had all sorts of services running on it. I removed the trojan'd ssh, asked the engineer if he thought there might be an issue, he said he didn't believe so. So I left it, until a few weeks later that trojan'd ssh came back. And at that point I shut the ethernet interfaces on the box off until it could be retired. There was no technical reason that it could not run behind the load balancer.

If you really need a front end firewall, consider a load balancer that has such functionality built in, because at least you have the ability to decrypt incoming SSL traffic and examine it, something very few firewall or IDS/IPS systems can do (another approach some people use is to decrypt at the load balancer than mirror the decrypted traffic to the IDS/IPS, but that is less secure of course).

It really does kind of scare me though that people seem to blindly associate a firewall with security, especially when it's a web server that is running. Now if those web servers were running RPC services and were hacked that way, a firewall very likely could of helped.

One company I worked at, my boss insisted we have firewalls in front of our load balancers, I couldn't convince him otherwise, so we deployed them. And they worked fine(for the most part). But the configuration wasn't really useful at all, basically we had a hole open in the firewall that pointed to the load balancer, which then pointed to the back end systems. So the firewall wasn't protecting anything that the load balancer wasn't doing already, a needless layer of complexity that didn't benefit anyone.

Myself I'm not convinced they were compromised via an Apache web server exploit, maybe they were compromised via an application running on top of Apache, but these days it's really rare to break into any web server directly via the web server software(whether it's Apache, IIS or whatever). I suspect they still don't really know how they were compromised and some manager at Sony pointed to that outdated software as the cause just so they could complete their internal processes on root cause and move on. Find something to tell congress, anything that sounds reasonable!!

Tagged as: , , Comments Off