TechOpsGuys.com Diggin' technology every day

23Aug/13Off

The world is ending: overreactions to outages

TechOps Guy: Nate

Paraphrasing from CNBC yesterday:

OMG!! CARL ICHAN IS TWEETERING ABOUT APPLE -- NASDAQ IS DOWN -- PEOPLE CAN'T TRADE ON THIS NEWS!!

Let me preface this a bit further and say in the line of work that I am in I have been on the receiving end of so many outages of various types ... some of them really nasty lasting hours, even down for multiple days, some involving some big data losses, many had me up for 20-30+ hours straight. Some of the most fun times I've had have been during big outages. Finally, some excitement!

My favorite outage that I can recall was one at AT&T about nine years ago. They were doing a massive migration to a new platform to support number portability, among other things. So they asked us to hold transactions in our queues while they were down for ~6-8 hours (the company I was at handled most of the mobile e-commerce for them at the time). So we did. 8 hours passed.. 10... 12... 16.. still down. No ETA .. It wasn't a huge deal for us for the first day, it became somewhat troublesome by the 3rd day as these queues were in memory and we had hard limits on memory(32-bit). But the folks in the AT&T stores were really hurting as they could not provision any new phones, all new orders had to be done on paper, then input into the computer system later. I forgot how long the outage was total I think around 4 days though. I looked at the whole situation and couldn't help but laugh. Lots of laughter. 8 hours to 4 days.. thousands of orders being placed via paper, by one of, if not the largest telcos in the world.

So I think I have a better perspective on this sort of thing than those less technical folk who freak out about stuff like the NASDAQ outage yesterday.

Taking NASDAQ specifically it was pretty absurd to see the whole situation unfold yesterday (I worked from home so saw the full thing end to end on CNBC). People coming on air and saying how they were frustrated that NASDAQ wasn't giving any information as to what was going on, speculation about complications being a public company and an exchange at the same time (and disclosure requirements etc). Then a bigwig comes on, Harvey Pitt, a former SEC chairman and just seems to ream NASDAQ, saying how it's totally unacceptable that they are down, there should be heavy fines and zero tolerance.

Come on folks - get a grip. It's a stock exchange. It's not a 911 system. People aren't going to die. If your system is so fragile that it can't survive a few hours of downtime on an exchange and can't tolerate a little volatility then it's your system that needs to be fixed.

You don't have control over the exchange, or the internet peering points between you and them(or your broker and them), there's so many points of failure that you should have a more robust system, the exchanges I have no doubt are incredibly complex, convoluted and obscure things that are constantly under attack by people trying to get trades through as quickly as possible, like those folks that manipulate the market.

Even the experts seem to be moving too fast, just barely a year ago Knight Capital lost $400 million in a matter of minutes due to a software bug. They later were forced to sell the company. More recently the almighty Goldman Sachs did something similar, last I saw they were hoping they would only lose $100 million as a result of that error.

Slow down, take a break. Things are moving too fast. I see people on CNBC constantly argue that the markets are really important because so many people have 401ks, IRAs etc. But reality doesn't agree with them. I don't recall the specific stat but I've heard it tossed around a few times something along the lines of 85% of stocks are owned by 5% of the population.

Another stat I've heard tossed around is  ~80% of the transactions these exchanges get today are from high frequency traders. So if HFT somehow goes away then these exchanges are in trouble revenue wise.

Those two stats alone tell me a lot about the state of the markets. I'm no financial expert obviously but I have watched CNBC a lot for many years now (going back to at least 2007) on a daily basis (RIP Mark Haines). I am often fascinated by the commentary, and the general absurdities of the market structure in general(I find in general it's more comedy than anything else). There's been very little investing going on for a very long time. Really the stock markets in general are outright gambling. Stocks rarely move on fundamentals anymore(not sure when they last did) it's all buzz, and emotions.

It's no wonder so many startups aren't interested in going IPO, and some other big established brands are wanting desperately to go private. To get away from activist investors, and the overall pressures to run your company in a pro-market fashion rather than what's best for the long term health of the company itself (and thus long term shareholders).

If your that dependent on market liquidity (e.g. the schemes folks like Lehman Brothers were doing rolling over their financials every night) - your doing something very wrong, and you deserve to get burned by it.

These exchanges are closed for upwards of 16 hours a day (and closed weekends and holidays!), there is some limited after hours trading, and some stocks trade on other exchanges as well, but the relative liquidity there is small.

This goes beyond NASDAQ of course and to outages in general. Whether it was the recent Google, or Amazon, or Microsoft, or whatever outages recently or others(I suffered through two yesterday myself that were the result of a 3rd party, one of which was literally minutes apart from when NASDAQ went down..).

So chill out. Fix the problem, don't rush or you might make a mistake and make things worse. Get it right, try not to let that particular scenario happen in the future.

It's just a website, it's just a stock exchange. It's not a nuclear reactor that is on the verge of melt down.

Breathe. The world isn't going to end because your site/business happens to be offline for a few hours.

Tagged as: 2 Comments
17Aug/13Off

Yahoo! yanks! website! of! person! who! documented! his! suicide!

TechOps Guy: Nate

The only thing technical related to this is the fact that Yahoo! yanked the guy's site. I suppose I can understand why, but I am very glad that the site(at least for the moment) lives on at a mirror.

I just came across this on slashdot. The discussion wasn't all that interesting but I have been reading a mirror of the website, and have to say it is quite an amazing write up.

I felt this guy took the time to think and write about his thoughts and did the world a favor in showing his state of mind. So the least I could do is read it - and perhaps comment on it a bit (with whatever respect I can give).

From what I can tell he committed suicide because he felt his mind was going, he was no longer (as) productive to society as he wanted to be, and he had a very negative outlook on near term civilization as we know it.

He took his life two days ago, on his 60th birthday in a police parking lot with a self inflicted gunshot wound to the head.

I have no idea who Martin Manley was but it was very interesting to see his line of thinking.

Some good quotes

I began seeing the problems that come with aging some time ago. I was sick of leaving the garage door open overnight. I was sick of forgetting to zip up when I put on my pants. I was sick of forgetting the names of my best friends. I was sick of going downstairs and having no idea why. I was sick of watching a movie, going to my account on IMDB to type up a review and realizing I've already seen it and, worse, already written a review! I was sick of having to dig through the trash to find an envelope that was sent to me so I could remember my own address - especially since I lived in the same place for the last nine years!

[..]

I didn’t want to die alone. I didn’t want to die of old age. I didn’t want to die after years of unproductivity. I didn’t want to die having my chin and my butt wiped by someone who might forget which cloth they used for which. I didn’t want to die of a stroke or cancer or heart attack or Alzheimer’s. I decided I was gettin’ out while the gettin’ was good and while I could still produce this website!

He does mention a life insurance policy that expires next year, and he wouldn't of been able to afford to renew it.  So that money can go to the folks he cares about.  Though I thought most, if not all such policies excluded suicide. I am not sure though, never looked into it. He seems intelligent enough that he would of known the details of the policy he had.

I felt pretty good about being prepared for economic collapse – the primary reason being all the gold and silver I owned. But, then one day I realized that all the gold and silver and guns and ammo and dried food and toilet paper in the world wouldn’t prevent me from seeing the calamity with my own eyes - either ignoring other's plight or succumbing to it. And, that’s something I decided I simply was not willing to live through.

Right with him there, except for the fact about being prepared. I acknowledged a long time ago that there's no point in trying to prepare for such an event, the resources required would be pretty enormous. My best friend(who is reading this, HI!!!)  has told me on a couple of occasions to go live with him in a cabin in the woods, live off the land(in the event of total collapse).. Not feasible for various reasons I don't want to get into here.

But, if you plan to stick around, then you better plan to watch an economic collapse that will be worse than anything you can imagine.

It's frustrating to me to see all of our leaders, whether it is in corporations or government show such, I'm not sure what the words are other than to call it something like false confidence.  Hiding the truth because sentiment is such an important factor of the economy. It's everywhere, the more I see folks talk the more I see in most cases they really have no idea what they are doing, they are just hoping it works out.

The more I learn the more I realize how young of a civilization we really are and how little we actually know.

What pisses me off more than anything is the system in place that tries to educate us so we think we know. So we have faith in those that are making the decisions.

I'm the first one to admit I don't know what the answers are(macro global economic/political type things) -- but I'm also (one of)the first ones to admit that uncertainty in the first place, which would probably make me a bad leader. I can't portray confidence because I don't have confidence(in that stuff anyway, I believe I do portray confidence when it comes to the tech things I work with), I do have honesty, which is what this Martin fellow seems to have a lot of as well. Most people don't want the truth, they just want to feel good.

One of the only videos I ever uploaded to Youtube was this, which is a good illustration from the corporate side of things. There is another bit (haven't been able to find it last I checked) which showed the same sort of thing from our previous president where they walked through his various descriptions of the impending economic downfall. From storm clouds to whatever it was in the end.

I'm in complete agreement with this Martin guy though, what we experienced in 2008-2010 or whatever is nothing compared to what is coming. When that is exactly I'm not sure, it seems folks like the Fed etc seem to be able to pull rabbits out of their hats to drove the ponzi scheme just a bit further. My general expectation is within the next two decades, and I think that is probably a conservative estimate.

It is unfortunate that the topic of suicide is amongst those topics that are considered taboo. People don't talk about it. The common theme is often mention the word and your deemed crazy and they want to lock you up in a padded room, fill you with meds until you conform, or die in the process.

There was a news report on NBC that I saw last year about a facility(hospital) where they assist people and their families to prepare for when that day will come. People perhaps like Martin who don't want to live out their lives as a burden to others, unable to mentally and/or physically perform things. I thought it was pretty amazing to see. They go through tons of questions with the patient about scenarios and what do do with those scenarios. So when the time comes there is no doubt. There won't be some random family member saying KEEP THEM ALIVE I DON'T CARE IF IT COSTS A MILLION $.

As with many topics, there was (in retrospect, it really didn't hit me until a few years ago) an excellent Star Trek: The Next Generation episode on this very topic. It was called Half a Life from Season 4 (1991). At the time when I saw it, I suppose you could say I didn't understand it, as I found the episode quite boring (well the special effects in the beginning were pretty neat). But as the health care debate picked up a few years ago I realized that episode told an interesting story that I never bothered to realize until that time.

Martin lists reasons he considered not to commit suicide (paraphrasing(?) them briefly, see the site for more details):

  1. Loved ones - obvious, people will miss you. More importantly perhaps is if you are in a situation where you are supporting someone else and they are dependent upon you. Martin was not in this situation
  2. Want to see the future. Live out retirement, travel the world perhaps, read books, play backgammon, look at granny porn (my grandfather did this a lot in the years before he died, honestly I did not know such porn existed until my sister+mother told me)
  3. People want to accomplish as much as they can in their lives and they don’t want to run out of time before they do it. Of course, for people who think that way, they never fulfill all those accomplishments anyway and they never will. So, the only thing to do is keep chasing them until you die.
  4. The last major reason I thought of for why people want to live indefinitely is the whole notion of leaving a legacy.

More quotes..

I once had a quasi bucket list when I was about 22 – things to accomplish by the time I was 30. When 30 came around and I hadn’t accomplish them, I decided the bucket list idea was stupid.

[..]

There will always be reasons to want to stay alive another year or five years or 10 years. It wouldn’t have mattered how long I lived, there would have been hundreds or thousands of itches to scratch!

[..]

I could take pride in the fact that I wasn’t going to be sucking on the nipple of the federal debt by taking social security and medicare. When the US economy collapses, it won't have been me that contributed to taking it down.

Here he touches on life insurance again, I guess they do pay out for suicide -

Another reason why age 60 is ideal is that my life insurance expires next year and I would not be able to afford to get new insurance without paying a ton. And, it requires two years of waiting - once you get insurance - before you can commit suicide and still have the beneficiaries receive the death benefit.

Holy sh*t, he even brings up the aforementioned Half a Life episode of TNG. He devotes several paragraphs to it!

Then he goes into how he did it - a gun

One of the problems with shooting oneself is the obvious mess. I thought about that a lot. I didn’t want anyone I knew discovering my body and I didn’t want to make a mess in the house – something my sister or my landlord would have to deal with. No way.

[..]

I finally decided the best way to do it would be at 5AM on August 15, 2013 at the far southeast end of the parking lot at the Overland Park Police Station. If everything worked out right – and I’m sure it did, I called 911 at 5AM. I told them “I want to report a suicide at the south end of the parking lot of the Overland Park Police Station at 123rd and Metcalf. Bang.”

He left a note on himself which in part read

“I committed suicide of my own free will. I am not under the influence of any drugs. I am sorry for your inconvenience! You will be contacted within a matter of hours by my sister. She will find out about this by an overnight letter and/or email I sent to her which she will get this morning. In it, I explain the exact location where I shot myself and gave her your phone number. At that time, she will tell you who I am. If you discover who I am prior to her call, please do not contact her. I do not want her (or anyone else I sent letters to overnight) to find out about it from you. I want them to find out about it from me. Thank you!

Another quote

The act of suicide can be horrible for those left behind. I couldn’t control the fact of the matter, but I could control the circumstances. I believe the way I did it, coupled with the overnight letters/emails and this web site, is the best I can do to mitigate the hurt.

He wrote a bunch more but the most interesting stuff I suspect was in the first few pages that I read (up until the "Gun Control" topic which is only semi related).

It is fascinating to me to see the level of thought and analysis that went into his decision. Whether you agree with the decision or not is up to you, but to me I think the point is it was his decision. Call him greedy, or selfish or crazy or whatever, but I firmly believe that having the "freedom" (though I think it is illegal(hence the quotes), which is crazy) to make this decision and follow through on it is an important right to have.

Now if you cause major harm as a result of your action (say perhaps drive down the wrong direction on a freeway) then that's a different scenario.  Martin was incredibly thoughtful in how he handled the whole situation, and for that I ..well I don't really know how to put that into words.

If you are someone who would want to rob someone else of this right then I'd say it is you who are greedy, selfish or crazy. The only exception would be again if you were somehow economically dependent upon them.

While I understand Yahoo! decision to yank the site, as it probably was against their TOS - I only wish Martin would of chosen a better place to host the site! Fortunately there is a mirror, I'll probably snag a copy of it myself just in case.

I don't consider myself a very emotional person, reading his writing really did not evoke any emotional response. I went to my first(thus far, only) funeral almost two years ago now for a cousin of mine who also committed suicide(also via self inflicted gunshot). I hadn't seen him since the late 80s, I had no emotional response for that either. I did feel bad for not "feeling" bad though, as strange as that may sound to write. I sat through the funeral as his loved ones and friends told stories about him and stuff for a few hours. It was an interesting experience. I'm sorry he is gone but from what I knew of the broader situations the family had experienced over the past few decades I can totally understand his decision. Anyway that is sort of off topic.

I hope the folks that where closest to Martin understand, accept, and most importantly support his decision.

I suppose the obvious thing to say here is may he rest in peace.

Any further discussion on the topic I am more than happy to talk about off line.

Tagged as: No Comments
5Jun/13Off

Real life Sim City

TechOps Guy: Nate

[WARNING: Non technical content directly ahead]

I've looked at Google maps a lot over the years, but don't remember ever seeing something quite like this. I played tons of Sim City many years ago and when I saw this I was immediately reminded of Sim City. It just seems so ..familiar.

I'm planning on staying at a hotel in this town in Nevada in a couple of weeks to visit a friend who is coming in from out of town(in case you were wondering how I stumbled upon this).

This first picture reminds me of many times when I would build out a neighborhood in Sim City with the roads, zone it with light (or medium) residential, perhaps  put neighborhood school near by - then watch the houses pop up one by one:

Real life Sim City Part 1

You can see a few individual houses here and there, and it's pretty easy to make out what look a lot like Sim City zoned plots of land(semi square shaped), obviously with a bunch of roads that are already complete. For the most part very clean empty plots of land. Much different than what I have seen many times in the past where perhaps there is a big real estate project under development and many houses are being built simultaneously with the road being laid out.

There is another part of the town that is quite similar, again eerily reminds me of Sim City:

Sim City in real life part 2

In this case I'm again reminded of some low density residential, along with a park in the middle(well in this case the other half of the middle is not yet laid down (not in the picture above, see the google maps link). The plots are so uniform, the houses remind me so much of Sim City.

Tagged as: No Comments
9May/13Off

Lost almost all respect for Consumer Reports today

TechOps Guy: Nate

I was getting my morning dose of CNBC this morning(I'm not an investor - I watch CNBC purely for entertainment purposes) when news came over the wire that Tesla had gotten a 99/100 rating on their ultra luxury green car by Consumer Reports.

I watched the interview with the guy at Consumer Reports and I was shocked, and I still am. A bit disgusted too.

Let me start out by saying I have no issues with the car itself. I'm sure it's a fine automobile. My issue is with the double talking standard of Consumer Reports in this particular case.

(forgive me the quotes will not be precise, see the video link above for the full interview)

The guy who wrote this report starts off by saying it's better than the other $90,000 cars in it's price range... (also goes on to say it's better than pretty much ANY car they have EVER EVER EVER TESTED -- not just better than any other electric car -- ANY CAR)

..but..

It is an electric car, while it has a long range compared to other electric cars, I can take a Toyota Corolla and drive to Cleveland from New York  -- I can't do that in this car yet.

You can only go about 200 miles before charging it up, that is a severe limitation. (those last two are his words)

CNBC goes on to quote him as saying If you leave it unplugged, you experience what you describe as a parasitic loss of energy(Consumer Report's words) that amounts to 12-15 miles per day, and asks him on that topic. He responds -

The concern is this is a new vehicle from a new automaker and there's going to be growing pains. If you're really looking for something to be trouble free off the bat - look elsewhere (his words too!)

I'm really glad I saw the interview. My disgust here is not with the car. I have no doubt it is a fine car! I think many people should go buy it. But if these sorts of flaws knock only a single point off the score ....that just seems wrong. Very wrong.  Especially the last bit -- if you want something to be trouble free look elsewhere -- for something that got rated 99 out of 100!!!

He goes on to talk about how it takes 30 minutes to charge the battery to half strength at one of Tesla's charging stations. He thinks people would be happier (duh) if they could fully charge it in four minutes.

CNBC half seriously asked him if you could charge a Tesla from a hotel room power outlet. Consumer reports guy said yes but it would take a VERY long time.

People buying this $90k car obviously are not concerned about the price of gas. It's really more for the image of trying to show your green than anything else, which is sad in itself(you can be more green by buying carbon offset credits - folks that can afford a $90k car should have no trouble buying some). But that's fine.

Again my issue isn't with the car. It's with the rating. Maybe it should be a 75, or an 85. I don't know. If it was me I'd knock at least 20 points off for the lack of range and lack of charging stations alone. Now if gas was say fifteen dollars a gallon I could see giving it some good credit for saving on that.

I think what Tesla is doing is probably a good thing, it seems like decent technology, good (relative to other electric vehicles) range. Likely can't take it on a road trip between SFO and Seattle any time soon..

I have relied at least in part on Consumer Reports for a few different purchases I have made (including my current car - which if I recall right Consumer Reports had no rating on it at the time it was too new. One of my friends just bought the 2013 model year of my car a few weeks ago).  It was extremely disappointing to see this result today. Maybe I should not of been surprised. I don't pay too close attention to what Consumer Reports does, I check them usually once every couple years at most. This may be par for the course for them.

Internet tech review sites have often had a terrible reputation for giving incorrect ratings for some reason or another (most often it seems to be because they want to keep getting free stuff from the vendors). I had thought(hoped) Consumer Reports was in a sort of league by itself for independent reporting.

I think at the end of the day the rating doesn't matter - people who are in the market for a $90k car will do their homework. It just sort of reeks to me of a back room deal or perhaps a bunch of hippies over at Consumer reports dreaming of a future where all vehicles are electric, never mind the fact the power grid can't handle it. (I can hear the voice of Cartman in the back of my head - Screw you hippies!). Tesla wants all the positive press they can get after all.

So let's see. We have a perfect score of 99/100 along which we have the words severe limitation, parasitic loss of energy, and look elsewhere if you want a trouble free experience ........

I'll say it one more time - my problem is with the perfect rating of 99/100 not with the car itself.

Tagged as: 2 Comments
1Apr/13Off

Public cloud will grow when I die

TechOps Guy: Nate

El Reg a few days ago posted some commentary from the CTO of Rackspace

Major adoption of public cloud computing services by large companies won't happen until the current crop of IT workers are replaced by kiddies who grew up with Facebook, Instagram, and other cloud-centric services

Which is true to some extent -- though I still feel the bigger issues with the public cloud are cost and features. If a public cloud company can offer comparable capabilities vs operating in house at a comparable (or less - given the cloud company should have bigger economies of scale) cost then I can see cloud really taking off.

As I've harped on again and again - one of the key cost things would be billing based on utilization, not based on what is provisioned (you could have a minimum commit rate as is often negotiated in deals for internet bandwidth). But if you provision a 2 CPU VM with 6GB of memory and 90% of the time it sits at 1% cpu usage and 1GB of memory then you must not be charged the same if you were consuming 100% cpu and 95% memory.

Some folks think it is a good idea to host non production stuff in a cloud and host production in house -- to me non production is where even more of the value comes from. Most of the time the non production environments(at least the companies I have worked at in the past decade) operate at VERY low utilization rates 99.9% of the time. So they can be over subscribed even more. At my organization for example we started out with basically two or three non production environments, now we are up to 10, the costs to support the extra 7-8 were minimal(relative to hosting them in a public cloud). For the databases I setup a snapshot system for these environments so not only can we refresh the data w/minimal downtime to the environments(about 2 minutes/ea vs/ full day/ea) but each environment typically consumes less than 10% of the disk space that would normally be consumed had the environment had a full independent copy of the data.

Another thing is give the customers the benefit of things like thin provisioning, data compression, deduplication. Some work loads behave better than others, present this utilization data to the customer and include it in the billing. Myself I like to provision multi TB volumes for almost everything, and I use LVM to restrict their growth. So if the time comes and some volume needs to get bigger I just lvextend the volume and resize the file system(both are online operations), don't have to touch the hypervsior, the storage, or anything. If some application may need a massive amount of storage (have not had one that did yet that used storage through the hypervisor) -- as in many many TB -- then I could allocate many volumes at once to the system, and grow them the same way over time. Perhaps a VM would have 2 to 10TB of space provisioned to it but may only use a few hundred gigs for the next year or so -- nothing is wasted, because the excess is not used. There's no harm in doing that.  Though I have not seen or heard of a cloud company that offers something like this. I think a large chunk of the reason is the technology doesn't exist yet to do it for hundreds or thousands of small/medium customers.

Most important of all - the cloud needs to be a true utility - 99.99% uptime demonstrated over a long period of time. No requirements for "built to fail", all failures should be transparent to the end user. Bonus points if you have the ability to have VMware-style fault tolerance (though something that can support multi CPUs) with millisecond fail over w/o data loss.   It will take a long time for the IaaS of the world to get there, but perhaps SaaS can be there already. PaaS I'm not sure, I've never dealt with that though. All of the major IaaS companies have had large scale outages and/or degraded performance.

The one area where public cloud does well - is the ability to get something from nothing up and going quickly, or perhaps up and going in a part of the country or world which you don't have a facility.  Though the advantage there isn't all that great. Even at my company back when we were hosted at Amazon on the east coast. The time came to bring up a site for our UK customers and we decided to host it on the east coast because the time frame to adapt everything(Chef etc) to work properly in another Amazon region was too tight to pull off. So we never used that region. Eventually we provisioned real equipment which I deployed in Amsterdam last summer to replace the last of our Amazon stuff.

Another article on a similar topic, this time from ComputerWorld, which noted the shift from in house data centers to service providers, though it seems more focused on literally in house data centers (vs "in house" with a co-location provider). They cite lack of available talent to manage these resources. These employees would rather work for a larger organization with more career opportunities than a small shop.

I'm sort of the opposite -- I would not like to work for a large company of any kind. Much prefer small companies, with small teams. The average team size I have worked in since 2006 has been 3 people. The amount of work required to maintain our own infrastructure is actually quite a bit less than managing cloud stuff.

I guess I am the exception rather than the rule again here. I had my annual review recently and in it I wrote there was no career advancement for me at the current company, I had higher growth expectations of the company I am at -- but I am not complaining. I'll admit that the stuff I am doing now is not as exciting as it has been in the past. I'm fairly certain we could not hire someone else in the team because they would get bored and leave quickly.  Me -- at least for now -- I don't mind being bored. It is a good change of pace after my run in the trenches along the front lines of technology from 2003-2011. I could do this for another year I imagine (if not longer).

As I watch the two previous companies I worked for wither and die slow deaths (and the one before them died years ago -- so basically all the jobs I had from 2006-2011 were at companies that are dead or dying) it's a good reminder to me to be thankful for where I am at. Still a small growing company with a good culture, good people, and everything runs really really well (sometimes so well it sort of scares me for some reason).

Another good reminder is I had lunch with a couple of friends while up in Seattle -- they work for a company that has been on it's death bed for years now. I asked them what is keeping the company going and they said hope (also never knew why they stuck around for as long as they have).  Or something like that. Not long after I left the company laid off a bunch of folks (they were not included in the layoff). The company is dieing every bit as much as the other two companies I worked for. I guess the main difference is I decided to jump ship long ago while they stuck it out for reasons unknown.

Time to close techopsguys?

I apologize again for not posting nearly as much as I used to -- there just continues to be a dearth of topics and news that I find interesting in recent months. I am not sure if it is me that is changing or if things have really gotten boring since the new year.  I have contemplated closing the blog entirely, just to lower people's expectations(or eliminate them) about seeing new stuff here.I've poured myself out all over this site the past few years and it's just become really hard to find things to write about now. I don't want the site to turn into a blog that is updated a couple times a year.

So I will likely close it in the coming months unless the situation changes. It has been a good run, from an idea from my former co-workers that I thought I'd be a minor contributor on to a full fledged site where I wrote nearly 400 articles, and a few hundred thousand words. Wow that is a lot.. My former co-workers bailed on the site years ago citing lack of time.  Time is certainly something I have what I have more is lack of things to write about.

I've had an offer to become an independent contributor to our friends over at El Reg - something that sounded cool at first, though now that I've thought about it I am not going to do it.  I don't feel comfortable about the level of quality I could bring (granted not all of their stuff is high quality either but I tend to hold myself to high standards). Being a personal blog I can compromise more on quality, lean into more of my own personal biases, I have less responsibility in general.

I have seen them take on a couple other bloggers such as myself in recent months and have noticed the quality of their work is not good.  In some cases it is sort of depressing(why would you write about that?????????)  That sort of stuff belongs on a personal blog not on a news site.

I'll have to settle for the articles where they mentioned my name in them, those I am still sort of proud of for some reason :)

 

Tagged as: 4 Comments
30Nov/12Off

Oh how I love the spin

TechOps Guy: Nate

So last night one of my friends sent me a link to an article saying that one of my former employers (whom I left more than two years ago) was closing one of their main business units and laying off 33 employees.

I'm quite out of touch with what is going on over there from a business standpoint, though I do know there's been a ton of attrition since I left (the team I was on has turned over entirely at least twice, and has had at least 3 different directors to lead it). I was really frustrated in 2010 which some of my posts at the time were pretty obvious, so I got out.

A few days after I left another guy started (he wasn't supposed to replace me but rather work on a project that I told the VP I refused to be a part of). I interviewed him even a few weeks before I left.  He tried, hard I imagine. I haven't talked to him but have talked to others who have - he left almost a year ago for similar reasons that I left - management. I've heard tons of stories about what's gone on there since I left, most of them very funny (wouldn't be if I was still there though). Sometimes you can't help but laugh.

The person who was in my position before me left for the same reason - these days I am VERY careful about finding management that is good to work with/for, it's more important than any other aspect of the job for me - I learned the hard way.

I can't count how many companies I have talked to or how many stories I have heard that have people being driven away by management decisions. It's just sick, it really is. I've talked to multiple companies where entire teams have turned over in very short periods of time (at least five in Seattle alone). Hell, I've been part of two of them at consecutive companies myself!

At one point this business unit that they are closing was responsible for a large portion of their revenue. I don't know where they stand today for revenues, I liked to tell people 2009 was the best year for the company - since then - not so much. Things really did look promising in 2009.....

I tried while I was there - I did - originally I was going to leave in December of 2008, barely 6 months after I started. But I decided to tough it out a bit more. I had some fun while I was there, management issues aside. Learned a few things, had some good experiences, met some cool people/friends. Considering it was literally across the street from my apartment at the time, it was difficult to beat that kind of commute.

They laid some folks off in early 2011 too (can't find news article) including two of my co-workers who both were begging to be laid off so they could collect unemployment.

Big surprise huh - anyway, I just look at the spin they put on the whole thing how it's a good thing for customers, good thing for everyone that they are making this decision.

I came across another article, which I found amusing:

To be honest, we’d never heard of AudienceScience before, but it is indeed a global digital marketing/tech firm that houses offices ranging from the U.S. to India. [..]

A web site seemingly focused on advertising agencies, hadn't even heard of them before.

I do own stock in the company - I only got it so I'd have a memento, I never bet anything on stock options.

We now take you back to your regularly scheduled programming.

Tagged as: No Comments
15Jun/12Off

Life extending IPv4 Patent issued

TechOps Guy: Nate

I've worked at a few companies over the years, more than one of them have crashed and burned but one in particular has managed to hold on, against the odds. Don't as me how because I don't know. Their stock price was $0.26 more than a decade ago (I recall my opening option price was in the $5 range back in the Summer of 2000).

CNBC Screen shot from a long time ago, it must of been good to be a treasury investor then look at that yield!

They didn't get into the patent game until years after they laid my friends and I off in 2002. The primary business model they had at the time, was making thin client software their main competition was the likes of Citrix and stuff. I worked on the side that made the Unix/Linux variant being a IT support/system admin for various Linux, Solaris, HPUX, Tru64 and AIX systems along with a few select Windows NT boxes. One of my long time friends worked on the Windows side of things at the other end of the country. The company acquired that technology from Corel I believe in the late 90s and still develop that product today. I'm not sure whether or not the Unix product is developed still for a long time they just had a single developer on it.

Anyways I write this because I was browsing their site yesterday while talking to that friend on the phone, and it turns out they were granted a groundbreaking new patent for cloud computing a couple of months ago. Though I think you'll agree that it's much more applicable towards extending the life of IPv4, without this technology we would of ran out of IPs a while ago.

U.S. Patent 8,117,298, issued February 14, 2012, is directed towards an easily configurable Web server that has the capability to host (or provide a home for) multiple Web sites at the same time. This patent helps small companies or individuals create the same kind of Web presence enjoyed by larger companies that are able to afford the cost of multiple, dedicated Web server machines.

This patent protects GraphOn's unique technology for “configuring” a Web server that has the capability to host multiple Web sites at the same time – called “multi-homing”. This multi-homing capability of the Web server provides the appearance to users of multiple distinct and independent servers at a lower cost.
Functionally, a multi-homed Web server consists of, in effect, multiple virtual Web servers running on the same computer. The patent claims a design in which features can be readily added to one or more of the virtual servers. For example, a new software module having additional features or different capabilities might be substituted for an existing module on one of the virtual servers. The new features or capabilities may be added without affecting any other of the virtual servers and without the need to rebuild the Web server.

You can see the uses for this patent right? I mean pretty much everyone out there will immediately want to step in and license it because it really is groundbreaking.

Another thing I learned from the patent itself which I was not aware of is that most web servers run under inetd -

Webs servers, most of which are written for UNIX, often run under INETD ("eye-net-D"), an Internet services daemon in UNIX. (A daemon is a UNIX process that remains memory resident and causes specified actions to be taken upon the occurrence of specified events.) Typically, if more than sixty connection requests occur within a minute, INETD will shut down service for a period of time, usually five minutes, during which service remains entirely unavailable.

This is not the first ground breaking patent they've been issued over the years.

Back in 2008 they were issued

  • Patent 7,386,880 for some form of load balancing.
  • Patent 7,424,737 for some sort of bridging technology that converts between IP and non IP protocols (the example given is Satellite protocols).
  • Patent 7,360,244 for two-factor authentication against a firewall.

Back in 2007 they were issued

  • Patent 7,269,591 which talks about a useful business model where you can charge a fee to host someone's personal web site
  • Patent 7,269,847 which is a very innovative technology involving configuration of firewalls using standard web browsers.
  • Patent 7,249,376 which covers multi homed firewalls and dynamic DNS updates
  • Patent 7,249,378 which seems to be associating a dedicated DNS for users utilizing a VPN.

Unfortunately for Graphon they did not license the patent that allows them to display more than the most recent five years of press releases on their site so I tasked our investigative sasquatch to find the information I require to finish this post.  Most/all of the below patents were acquired with the acquisition of Network Engineering Software.

Harry, our investigative sasquatch

Back in 2006 they were issued

  • Patent 7,028,034 covers web sites that dynamically update and pull information from a database to do so. Fortunately for you non-profits out there the scope is limited to pay sites.

..in 2005

  • Patent 6,850,940 which I'm honestly surprised someone like Oracle didn't think of earlier, it makes a lot of sense. This covers maintaining a network accessible database that can receive queries from remote clients.

..And Waaay back in 2001

  • Patent 6,324,528 which covers something along the lines of measuring the amount of time a service is in use - I think this would be useful for the cloud companies too you need to be able to measure how much your users are using the service to really bill based on usage rather than what is provisioned.

I suppose I should of bought my options when I had the chance I mean if I had invested in the $5 option price I would be able to retire, well maybe not, given the stock is trading in the $0.12 range. I felt compelled to get this news out again so that the investors can wise up and see what an incredible IP portfolio this company has and the rest of the world needs to stand ready to license these key technologies.

Then I can go back in time and tell myself to buy those options only to come forwards in time and retire early. I'll publish my time travel documents soon, I won't ask you to license them from me, but you will have to stop at my toll both in the year 2009 to pay the toll. You're free to travel between now and Jan 1 2009 without any fees though, think of it as a free trial.

Tagged as: 2 Comments
6Apr/12Off

Flash from the past – old game review

TechOps Guy: Nate

I was talking to a friend that I've known for more than 20 years a few days ago (these posts really are making me feel old :) ) and we were talking about games, Wing Commander Saga among them and he brought up an old game that we tried playing for a while he couldn't remember the name, but I did. It was Outpost. A Sci-fi simcity or civilization style game from 1994. It has amazing visuals, being one of the earlier CDROM-based games. I bought it after seeing the visuals and really wanted to like it but no matter what I couldn't get very far without losing the game, no matter what I did I would run out of resources, air, food water, whatever it was, all my people would die and I would have to start again. Rinse and repeat a few times and I gave up on it eventually. I really liked the concept being a long time sci fi fan (well hard core sci fi fans would probably refer to me as a space opera fan).

ANYWAYS, I hadn't looked for anything about this game since well probably the mid 90s. I ran a basic search earlier today and came across this 10-minute video review from an interactive CDROM magazine published back in 1994. I had no idea how much of a mess the game really was, the review was incredibly funny to watch they rip the game apart. They are in awe of the visuals like I was but other than that the game was buggy and under finished. They make it sound like it was one of the most un finished games of all time. They talk about entire sections of the strategy guide that are completely left out of the game, functions that are mentioned that just don't exist. In-game artificial intelligence that gives absolutely worthless data, no explanation on how to plan to play the game up front. It's quite humorous! I'm going to go watch it again.

From Wikipedia

Initial reviews of Outpost were enthusiastic about the game. Most notoriously, the American version of PC Gamer rated the game at 93%, one of its highest ratings ever for the time. It was later made known that the reviewers had in fact played beta versions of the game, and had been promised certain features would be implemented, but never were.

[..]

Following the release of the game, the game's general bugginess and perceived mediocre gameplay, along with the lack of features described in most of the game's reviews and the game's own documentation led to a minor backlash against the computer game magazines of the time by consumers who bought the game based on their reviews.

Tagged as: No Comments
5Apr/12Off

Built a new computer – first time in 10 years

TechOps Guy: Nate

I was thinking about when the last time I built a computer from scratch this morning and I think it was about ten years ago, maybe longer - I remember the processor was a Pentium 3 800Mhz. It very well may of been almost 12 years ago. Up until around 2004 time frame I had built and re-built computers re-using older parts and some newer components, but as far as going out and buying everything from scratch, it was 10-12 years ago.

I had two of them, one was a socket-based system the other was a "Slot 2"-based system. I also built a couple systems around dual-slot (Slot 1) Pentium 2 systems with the Intel L440GX+ Motherboard (probably my favorite motherboard of all time). For those of you think that I use nothing but AMD I'll remind you that aside from the AMD K6-3 series I was an Intel fanboy up until the Opteron 6100 was released. I especially liked the K6-3 for it's on chip L2 cache, combined with 2 Megabytes of L3 cache on the motherboard it was quite zippy. I still have my K6-3 CPU itself in a drawer around here somewhere.

So I decided to build a new computer to move my file serving functions out of my HP xw9400 workstation which I bought about a year and a half ago into something smaller so I could turn the HP box into something less serious to play some games on my TV on (like WC: Saga!). Maybe get a better video card for it I'm not sure.

I have a 3Ware RAID card and 4x2TB disks in my HP box so I needed something that could take that. This is what I ended up going with, from Newegg -

Seemed like an OK combination, the case is pretty nice having a 5-port hot swap SATA backplane, supporting up to 7 HDDs. PC Power & Cooling (I used to swear by them and so thought might as well go with them again) had a calculator and said for as many HDDs as I had to get a 500W so I got that.

There is a lot of new things here that are new to me anyways and it's been interesting to see how technology has changed since I last did this in the Pentium 3 era.

Mini ITX. Holy crap is that small. I knew it was small based on dimensions but it really didn't set in until I held the motherboard box in my hand and it seemed about the same size as a retail box for a CPU 10 years ago. It's no wonder the board uses laptop memory. The amount of integrated features on it are just insane as well from ethernet to USB 2 and USB 3, eSATA, HDMI, DVI, PS/2, optical audio output, analog audio out, and even wireless all crammed into  that tiny thing. Oh! Bluetooth is thrown in as well. During my quest to find a motherboard I even came across a motherboard that had a parallel port on it - I thought those died a while ago. The thing is just so tiny and packed.

On the subject of motherboards - the very advanced overclocking functions is just amazing. I will not overclock since I value stability over performance, and I really don't need the performance in this box. I took the overclocking friendliness of this board to hopefully mean higher quality components and the ability to run more stable at stock speeds. Included tweaking features -

  • 64-step DRAM voltage control
  • Adjustable CPU voltage at 0.00625V increments (?!)
  • 64-step chipset voltage control
  • PCI Express frequency tuning from 100Mhz up to 150Mhz in 1Mhz increments
  • HT Tuning from 100Mhz to 550Mhz in 1Mhz increments
  • ASUS C.P.R. (CPU Parameter Recall) - no idea what that is
  • Option to unlock the 4th CPU core on my CPU
  • Options to run on only 1,2 or or all 3 cores.

Last time I recall over clocking stuff there was maybe 2-3 settings for voltage and the difference was typically at least 5-15% between them. I remember the only CPU I ever over clocked was a Pentium 200 MMX (o/c to 225Mhz - no voltage changes needed ran just fine).

I seem to recall from a PCI perspective, back in my day there was two settings for PCI frequencies, whatever the normal was, and one setting higher(which was something like 25-33% faster).

ASUS M4A88T-I Motherboard

Memory - wow it's so cheap now, I mean 8GB for $45 ?! The last time I bought memory was for my HP workstation which requires registered ECC - and it was not so cheap ! This system doesn't use ECC of course. Though given how dense memory has been getting and the possibility of memory errors only increasing I would think at some point soon we would want some form of ECC across the board ? It was certainly a concern 10 years ago when building servers with even say 1-2GB of memory now we have many desktops and laptops coming standard with 4GB+. Yet we don't see ECC on the desktops and laptops - I know because of cost but my question is more around there doesn't appear to be a significant (or perhaps in some cases even noticeable) hit in reliability of these systems with larger amounts of memory without ECC which is interesting.

Another thing I noticed was how massive some video cards have become, consuming as many as 3 PCI slots in some cases for their cooling systems. Back in my day the high end video cards didn't even have heat sinks on them! I was a big fan of Number Nine back in my day and had both their Imagine 128 and Imagine 128 Series 2 cards, with a whole 4MB of memory (512kB chips if I remember right on the series 2, they used double the number of smaller chips to get more bandwidth). Those cards retailed for $699 at the time, a fair bit higher than today's high end 3D cards (CAD/CAM workstation cards excluded in both cases).

Modular power supplies - the PSU I got was only partially modular but it was still neat to see.

I really dreaded the assembly of the system since it is so small, I knew the power supply was going to be an issue as someone on Newegg said that you really don't want a PSU that is longer than 6" because of how small the case is. I think PC Power & Cooling said mine was about 6.3"(with wiring harness). It was gonna be tight -- and it was tight. I tried finding a shorter power supply in that class range but could not. It took a while to get the cables all wrapped up. My number one fear of course after doing all that work, hitting the power button and find out there's a critical problem (bought the wrong ram, bad cpu, bad board, plugged in the power button the wrong way whatever).

I was very happy to see when I turned it on for the first time it lit up and the POST screen came right up on the TV. There was a bad noise comming from one of the fans because the cable was touching it, so I opened it up again and tweaked the cables more so they weren't touching the fan, and off I went.

First, without any HDs just to make sure it turned on, the keyboard worked, I could get into the BIOS screen etc. All that worked fine, then I opened up the case again and installed an old 750GB HD in one of the hot swap slots. Hooked up a USB CDROM with a CD of Ubuntu 10.04 64-bit and installed it on the HD.

Since this board has built in wireless I was looking forward to trying it out - didn't have much luck. It could see the 50 access points in the area but it was not able to login to mine for some reason, I later found that it was not getting a DHCP response so I hard wired an IP and it worked -- but then other issues came up like DNS not working, very very slow transfer speeds(as in sub 100 BYTES per second), after troubleshooting for about 20 minutes I gave up and went wired and it was fast. I upgraded the system to the latest kernel and stuff but that didn't help the wireless. Whatever, not a big deal I didn't need it anyways.

I installed SSH, and logged into it from my laptop,  shut off X-Windows, and installed the Cerberus Test Suite (something else I used to swear by back in the mid 00s). Fortunately there is a packaged version of it for Ubuntu as, last I checked it hasn't been maintained in about seven years. I do remember having problems compiling it on a 64-bit RHEL system a few years ago (Though 32-bit worked fine and the resulting binaries worked fine on 32-bit too).

Cerberus test suite (or ctcs as I call it), is basically a computer torture test. A very effective one, the most effective I've ever used myself. I found that if a computer can survive my custom test (which is pretty basic) for 6 hours then it's good, I've run the tests as long as 72 hours and never saw a system fail in a period more than 6 hours. Normally it would be a few minutes to a couple hours. It would find problems with memory that memtest wouldn't find after 24 hours of testing.

What cerberus doesn't do, is it doesn't tell you what failed or why, if your system just freezes up you still have to figure it out. On one project I worked on that had a lot of "white box" servers in it, we deployed them about a rack at a time, and I would run this test, maybe 85% of them would pass, and the others had some problem, so I told the vendor go fix it, I don't know what it is but these are not behaving like the others so I know there is a issue. Let them figure out what component is bad (90% of the time it was memory).

So I fired up ctcs last night, and watched it for a few minutes, wondering if there is enough cooling on the box to keep it from bursting into flames. To my delight it ran great, with the CPU topping out at around 54C (honestly have no idea if that is good or not, I think it is OK though). I ran it for 6 hours overnight and no issues when I got up this morning. I fired it up again for another 8 hours (the test automatically terminates after a pre defined interval).

I'm not testing the HD, because it's just a temporary disk until I move my 3ware stuff over.  I'm mainly concerned about the memory, and CPU/MB/cooling. The box is still running silent (I have other stuff in my room so I'm sure it makes some noise but I can't hear it). It has 4 fans in it including the CPU fan. A 140mm, a 120mm and the PSU fan which I am not sure how big that is.

My last memory of ASUS was running on an Athlon with an A7A-266 motherboard(I think in 2000), that combination didn't last long. The IDE controller on the thing corrupted data like nobody's business. I would install an OS, and everything would seem fine then the initial reboot kicked in and everything was corrupt. I still felt that ASUS was a pretty decent brand maybe that was just specific to that board or something. I'm so out of touch with PC hardware at this level the different CPU sockets,  CPU types, I remember knowing everything backwards and forwards in the Socket 7 days, back when things were quite interchangeable. Then there was my horrible year or two experience in the ABIT BP-6, a somewhat experimental dual socket Celeron system. What a mistake that was, oh what headaches that thing gave me. I think I remember getting it based on a recommendation at Tom's Hardware guide, a site I used to think had good information (maybe it does now I don't know). But that experience with the BP6 fed back into my thoughts about Tom's hardware and I really didn't go back to that site ever again(sometimes these days I stumble upon it on accident). I noticed a few minutes ago that Abit as a company is out of business now, they seemed to be quite the innovator back in the late 90s.

Maybe this weekend I will move my 3ware stuff over and install Debian (not Ubuntu) on the new system and set it up. While I like Red Hat/CentOS for work stuff, I like Debian for home. It basically comes down to if I am managing it by hand I want Debian, if I'm using tools like CFEngine to manage it I want RH. If it's a laptop, or desktop then it gets Ubuntu 10.04 (I haven't seen the nastiness in the newer Ubuntu release(s) so not sure what I will do after 10.04).

I really didn't think I'd ever build a computer again, until this little side project came up.

Tagged as: , No Comments
17Feb/12Off

Random Irony – MySQL wanting .NET

TechOps Guy: Nate

So I am here late at night waiting for Citrix to call me back about a issue on one of my Netscalers so I went to install an eval of the 3PAR System Reporter -- last time I used it it was windows-only, even though it was based off of Apache, MySQL and some CGI. Out of instinct I installed a windows VM to install this on and since it is an eval I am just going to put MySQL on the same VM because well our array is small and there just isn't enough data to tax it. Later I found out I can use Linux now! But for Linux it needs 32-bit, so for this eval I'll stick to windows since I'm almost done setting it up and I don't have 32-bit Linux handy at the moment.

I've never been a big fan of system reporter, it seems very much like the tools from EMC anyways, and I've heard similar about the tools from HDS. So I guess it is typical, which is probably why I am not fond of it. But installing it is faster and easier than getting my home grown stuff online. So in the meantime I'll use it till I get some spare time to install my custom stuff (which uses perl scripts, rrdtool and cacti to display the data - along with a lot of time to create the graphs).

So I was installing some things and went to install MySQL, much to  my surprise, it wanted .NET. Oracle owns MySQL, they own Java, MySQL is of course open source, and I'm sure it's got more deployments on Linux and stuff than Windows so it puzzles me after all that, that the MySQL folks go out of their way to use .NET for whatever they are using it for.

Thought it was funny anyways

Where's my NET?

Little update - got a cryptic error while trying to install system reporter.. so I guess here comes another support case for HP!

Tagged as: No Comments