TechOpsGuys.com Diggin' technology every day

Sony Compromised by Apache bug?

TechOps Guy: Nate

Came across an article from a friend that talks about how Sony thinks they were compromised.

According to Spafford, security experts monitoring open Internet forums learned months ago that Sony was using outdated versions of the Apache Web server software, which “was unpatched and had no firewall installed.”

The firewall part is what gets me. Assuming of course this web server(s) were meant to be public, no firewall is going to protect you against this sort of thing since of course firewalls protecting public web servers have holes opened explicitly for the web server so all traffic is passed right through.

And I highly doubt those Apache web servers had confidential data as the article implies, obviously that data was on back end systems running databases of some sort.

Then there are people out there spouting stuff on PCI saying the automated external scans should of detected they were running outdated versions of software. In my experience such scans are really not worth much with Linux, primarily because they have no way to take into account patches that are back ported to the operating system. I’ve had a few arguments with security scanners trying to explain how a system is patched because the fix was back ported but them not being able to comprehend that because the major/minor version being reported by the server has not changed.

Then there was the company I worked for who had a web app that returned a HTTP/200 for pretty much everything, including things like 404s. This tripped every single alarm the scanners had, and they went nuts. And once again we had to explain that those windows exploits aren’t going to work against our Apache Tomcat systems running Linux.

IDS and IPS are overrated as well, unless you really have the staff to watch and manage it full time. In all of the years I have worked at companies that deployed some sort of IDS (never IPS), I have seen it work, one time, back in I want to say 2002, I saw a dramatic upsurge in some type of traffic on our Snort IDS at the time from one particular host and turns out it had a virus on it. I worked at one company that was compromised at LEAST twice while I was there(on systems that weren’t being properly managed). and of course the IDS never detected a thing. Then that company deployed(after I left) a higher end hardware-based IPS, and when they put it inline to the network (in passive, not enforcing mode) for some reason the IPS started dropping all SSL traffic for no reason.

They aren’t completely useless though, they can help detect and sometimes protect against the more obvious types of attacks (SQL injection etc).  But in the grand scheme of things, especially when dealing with customized applications (not off the shelf like Exchange, Oracle or whatever), IDS/IPS and even firewalls provide only a tiny layer of additional security on top of good application design, good deployment practices(e.g. don’t run as root, disable or remove subsystems that are not used, such as the management app in Tomcat, use encryption where possible), and a good authentication system for system level access (e.g. ssh keys). With regards to web applications, a good load balancer is more than adequate to protect the vast majority of applications out there, it is “firewall like” as in it only passes certain ports to the back end systems, but (for higher traffic sites this is important) vastly outperforms firewalls, which can be a massive bottleneck for front end systems.

With regards to the company that was compromised at least twice, the intrusion was minor and limited to a single system, the compromise occurred because the engineer who installed the system put it outside of the load balancers, it was a FTP server, or was it a monitoring server, I forgot.  Because it needed to be accessed externally the engineer thought hey let’s just put it on the internet. Well it sat there for a good year or two, (never being patched in the meantime) before I joined the company, compromised in some fashion, and ssh was replaced with a trojaned copy (it was pretty obvious, I am assuming it was some sort of worm exploiting ssh). It had all sorts of services running on it. I removed the trojan’d ssh, asked the engineer if he thought there might be an issue, he said he didn’t believe so. So I left it, until a few weeks later that trojan’d ssh came back. And at that point I shut the ethernet interfaces on the box off until it could be retired. There was no technical reason that it could not run behind the load balancer.

If you really need a front end firewall, consider a load balancer that has such functionality built in, because at least you have the ability to decrypt incoming SSL traffic and examine it, something very few firewall or IDS/IPS systems can do (another approach some people use is to decrypt at the load balancer than mirror the decrypted traffic to the IDS/IPS, but that is less secure of course).

It really does kind of scare me though that people seem to blindly associate a firewall with security, especially when it’s a web server that is running. Now if those web servers were running RPC services and were hacked that way, a firewall very likely could of helped.

One company I worked at, my boss insisted we have firewalls in front of our load balancers, I couldn’t convince him otherwise, so we deployed them. And they worked fine(for the most part). But the configuration wasn’t really useful at all, basically we had a hole open in the firewall that pointed to the load balancer, which then pointed to the back end systems. So the firewall wasn’t protecting anything that the load balancer wasn’t doing already, a needless layer of complexity that didn’t benefit anyone.

Myself I’m not convinced they were compromised via an Apache web server exploit, maybe they were compromised via an application running on top of Apache, but these days it’s really rare to break into any web server directly via the web server software(whether it’s Apache, IIS or whatever). I suspect they still don’t really know how they were compromised and some manager at Sony pointed to that outdated software as the cause just so they could complete their internal processes on root cause and move on. Find something to tell congress, anything that sounds reasonable!!

   Send article as PDF to