TechOpsGuys.com Diggin' technology every day

Paraphrasing from CNBC yesterday:

OMG!! CARL ICHAN IS TWEETERING ABOUT APPLE — NASDAQ IS DOWN — PEOPLE CAN’T TRADE ON THIS NEWS!!

Let me preface this a bit further and say in the line of work that I am in I have been on the receiving end of so many outages of various types … some of them really nasty lasting hours, even down for multiple days, some involving some big data losses, many had me up for 20-30+ hours straight. Some of the most fun times I’ve had have been during big outages. Finally, some excitement!

My favorite outage that I can recall was one at AT&T about nine years ago. They were doing a massive migration to a new platform to support number portability, among other things. So they asked us to hold transactions in our queues while they were down for ~6-8 hours (the company I was at handled most of the mobile e-commerce for them at the time). So we did. 8 hours passed.. 10… 12… 16.. still down. No ETA .. It wasn’t a huge deal for us for the first day, it became somewhat troublesome by the 3rd day as these queues were in memory and we had hard limits on memory(32-bit). But the folks in the AT&T stores were really hurting as they could not provision any new phones, all new orders had to be done on paper, then input into the computer system later. I forgot how long the outage was total I think around 4 days though. I looked at the whole situation and couldn’t help but laugh. Lots of laughter. 8 hours to 4 days.. thousands of orders being placed via paper, by one of, if not the largest telcos in the world.

So I think I have a better perspective on this sort of thing than those less technical folk who freak out about stuff like the NASDAQ outage yesterday.

Taking NASDAQ specifically it was pretty absurd to see the whole situation unfold yesterday (I worked from home so saw the full thing end to end on CNBC). People coming on air and saying how they were frustrated that NASDAQ wasn’t giving any information as to what was going on, speculation about complications being a public company and an exchange at the same time (and disclosure requirements etc). Then a bigwig comes on, Harvey Pitt, a former SEC chairman and just seems to ream NASDAQ, saying how it’s totally unacceptable that they are down, there should be heavy fines and zero tolerance.

Come on folks – get a grip. It’s a stock exchange. It’s not a 911 system. People aren’t going to die. If your system is so fragile that it can’t survive a few hours of downtime on an exchange and can’t tolerate a little volatility then it’s your system that needs to be fixed.

You don’t have control over the exchange, or the internet peering points between you and them(or your broker and them), there’s so many points of failure that you should have a more robust system, the exchanges I have no doubt are incredibly complex, convoluted and obscure things that are constantly under attack by people trying to get trades through as quickly as possible, like those folks that manipulate the market.

Even the experts seem to be moving too fast, just barely a year ago Knight Capital lost $400 million in a matter of minutes due to a software bug. They later were forced to sell the company. More recently the almighty Goldman Sachs did something similar, last I saw they were hoping they would only lose $100 million as a result of that error.

Slow down, take a break. Things are moving too fast. I see people on CNBC constantly argue that the markets are really important because so many people have 401ks, IRAs etc. But reality doesn’t agree with them. I don’t recall the specific stat but I’ve heard it tossed around a few times something along the lines of 85% of stocks are owned by 5% of the population.

Another stat I’ve heard tossed around is  ~80% of the transactions these exchanges get today are from high frequency traders. So if HFT somehow goes away then these exchanges are in trouble revenue wise.

Those two stats alone tell me a lot about the state of the markets. I’m no financial expert obviously but I have watched CNBC a lot for many years now (going back to at least 2007) on a daily basis (RIP Mark Haines). I am often fascinated by the commentary, and the general absurdities of the market structure in general(I find in general it’s more comedy than anything else). There’s been very little investing going on for a very long time. Really the stock markets in general are outright gambling. Stocks rarely move on fundamentals anymore(not sure when they last did) it’s all buzz, and emotions.

It’s no wonder so many startups aren’t interested in going IPO, and some other big established brands are wanting desperately to go private. To get away from activist investors, and the overall pressures to run your company in a pro-market fashion rather than what’s best for the long term health of the company itself (and thus long term shareholders).

If your that dependent on market liquidity (e.g. the schemes folks like Lehman Brothers were doing rolling over their financials every night) – your doing something very wrong, and you deserve to get burned by it.

These exchanges are closed for upwards of 16 hours a day (and closed weekends and holidays!), there is some limited after hours trading, and some stocks trade on other exchanges as well, but the relative liquidity there is small.

This goes beyond NASDAQ of course and to outages in general. Whether it was the recent Google, or Amazon, or Microsoft, or whatever outages recently or others(I suffered through two yesterday myself that were the result of a 3rd party, one of which was literally minutes apart from when NASDAQ went down..).

So chill out. Fix the problem, don’t rush or you might make a mistake and make things worse. Get it right, try not to let that particular scenario happen in the future.

It’s just a website, it’s just a stock exchange. It’s not a nuclear reactor that is on the verge of melt down.

Breathe. The world isn’t going to end because your site/business happens to be offline for a few hours.