After some investigation with a developer we discovered it was an issue with Facebook (ugh, how I hate thee) and their code was breaking because Facebook was broken (they were not aware this would happen I am sure they will fix it going forward).

So while they are working to work around the issue I ran a search for it, and it seems to be a more wide spread problem caused by a Facebook software deployment done today, Friday at nearly 7PM!

F F S

My monitor first detected the failure at 6:49 PM so they weren’t even done deploying by the time it failed. It was intermittant for a few minutes then went hard down at around 7:11PM.

@$#$ facebook. Thanks for screwing me, and who knows how many others by deploying code on a Friday night.

Of course the developers on our end deserve some of the heat as well. But I am nit picking about code deployments on a Friday, not code bugs..

   Send article as PDF to

The performance results are certainly very impressive, coming in at over 520,000 IOPS  at a price of $6.92 per IOP. This is the sort of results I was kind of expecting from the Hitachi VSP a while back. IBM tested with 1,920 drives the same number as the 3PAR V800. They bested the 3PAR performance by a good 70,000 IOPS with half the latency on the same number of disks and less data cache.

The capacity numbers were, and still are, sort of difficult to interpret they seem to give conflicting information. IBM is using ~138TB of disk space to protect ~99TB of disk space. While 3PAR is using ~263TB of disk space to protect ~263TB of disk space. Both results say there is 30TB+ of “unused storage” in that protection scheme.

Bottom line is the IBM box is presented with roughly 280TB of storage, and of that, 100TB is usable, or about 35%. That brings their cost per usable TB number to $36,881/TB vs the 3PAR V800 which is roughly $12,872. The V800 I/O cost was $6.59, which IBM comes real close to.

IBM has apparently gone the same route as HDS in the only 3.5″ drives they support on their Storwize systems are 3TB SATA disks. They hamper their own cost structure by not supporting larger 3.5″ 15k RPM SAS disks, which just doesn’t make sense to me. There are 300GB 15k SAS drives out and Storwize doesn’t support those either(yet at least).

It took about five pages of scripting to configure the system from the looks of the full disclosure report.

Certainly looks like a halfway decent system. I mean if you compare it to the VSP for example it has the same array virtualization abilities with the SVC, it is sporting almost double the amount of disk drives, almost double the raw performance, configuration at least appears to be less complicated. It uses those power efficient 2.5″ disks just like the VSP. It also costs quite a bit less than the VSP both on per-IOP and per-TB basis. It also appears to have mainframe support for those that need that. From the looks of Seagate 15k RPM disks at least the 2.5″ drives have an average of 15% less latency for random reads and writes than their 3.5″ counterparts. I thought the difference might be bigger than that given how much less distance the disk heads have to travel.

If I was in the market for such a big system, these results wouldn’t lead me away from 3PAR, at least based on the pricing disclosed of each system (and level of complexity to configure). I was interviewing a candidate a few weeks ago and this guy had a strong storage background. Having worked for Symantec I think for a while he was doing some sort of consulting at various companies for storage. I asked him how he provisioned storage, what his strategies were. His response was quite surprising. He said usually the vendors come out, deploy their systems and provision everything up front, all he does is carve out LUNs and present them to users. He had never been involved in the architecture planning or deployment of a storage system. He acted as if what he was doing was the standard practice (maybe it is at large companies I’ve never worked at such an organization), and that it was perfectly normal.

But it certainly seems like a good system when put up against at least the VSP, and probably the V-MAX too.

I’ve always been interested in the SVC by itself, certainly seems like a cool concept, I’ve never used one of course but having the ability to cluster at that intermediate level(in this case a 8-node cluster which may be the max I’m not sure) and then scale out storage behind it. Clearly they’ve shown with this you can pump one hell of a lot of I/O through the thing. They also seem to have SSD tiering support built into it which is nice as well.

Hopefully HP can come up with something similar at some point, as much as they talk smack about the likes of SVC today.

   Send article as PDF to

I wanted to write about a sort of topic that I haven’t really written about before. It’s about a systems management tool named Chef from a Seattle-based company named Opscode. It’s supposed to be a next generation tool that is supposed to make your life easier, more advanced than older tools like Puppet and Cfengine.

I’ll start off by saying I have a very strong background in Cfengine, having had used it since late 2004, at three different companies. My techniques and approaches evolved significantly over the years, and my last deployment was quite good in my opinion considering I had to adapt an existing Cfengine deployment made by folks who didn’t know what they were doing into something that worked well, and doing so in a 4 nines environment. That was not easy, as you know one wrong command or config in one of these tools can wreck havoc as I know first hand. I grew to like Cfengine a lot, and there was really nothing that I needed it to do that it couldn’t do for me. I knew it’s limitations well and it was simple to use.

I was introduced to Chef in the summer of 2010 when I went to the headquarters of Opscode and met their senior staff including one of the co-founders I believe. They gave us their powerpoint presentation on what Chef was, how it worked, what it could do, why it exists.

It certainly came across as a very impressive tool, being able to do tons of things that Cfengine could not do, had a lot of concepts that sounded like they could be useful. At the same time however it looked incredibly complicated.

I raised my concerns with their senior staff on that very first day and we had about a 15 minute discussion on it. I’m not a programmer, nor do I ever intend to be. I have a very big line that I refuse to cross from scripting tools in perl & bash to help make my life easier to full on code. A developer at my company constantly jokes that I say I am not a programmer yet I come up with complicated regexes and scripts to do things they don’t understand how to come up with on their own.

They tried to re-assure me that learning Chef is no different than learning the syntax of an Apache configuration file, or DNS or something like that. I didn’t really buy it, but was still willing to give the tool a shot since it sounded like a nice level of systems management that you could achieve with it. I still joke with my co-workers and current boss( who was my boss at the time too) on this very topic, they all remember that conversation to this day.

Chef is written in Ruby, and is very Ruby-centric. I guess you could say I am very biased against Ruby given my past experience supporting Ruby (on Rails) applications.

So here I am, almost 18 months later and things haven’t changed much. My dislike of Ruby continues, and is perhaps even stronger now having used Chef.

My first chef implementation about a year ago was fraught with frustration at almost every turn. I could(and still can) see the promise in the tools it provides the user with but it’s just so difficult to work with especially coming from a Cfengine background(and lack of programming experience) that for my first iteration I dumbed it down a whole bunch, making the logic very Cfengine like, at least as much as I could. I didn’t use any data bags, any attributes, no templates, nothing like that. I had (and still have) a very hard time finding usable examples for many things in Chef. They have a big repository of sample cookbooks – but to me for the most part those are not usable, because while examples they don’t go into details as to specifically, literally what each line of code does. Chef apparently uses this for it’s template language, I looked at it a couple of times – and really I could not make heads or tails of it.

I like to tell people that Chef makes the easy things hard, and the hard things possible. It seems very clear to me that they attacked the hard things in system management first before addressing the easy things. I remember seeing something in their documentation around the concept of the holy grail in the single instance copy, which fits along those lines well. The idea is you have one small bit of code that can be adapted to (m)any environments and situations, using the templates to pull attributes and values from data bags or other sources to make something on the fly.

The concept is novel for sure, coming from a Cfengine background I am very used to duplicating config stanzas, for different environments, making static config files, one for each environment or something like that. I’ve been doing it so long it’s second nature.

Where the opscode folks and I seem to part ways is our priorities. Their priority is to turn the system management into code and automate it to the point where it scales to a million systems. Mine is less ambitious, I want it to be easy to manage and it can scale to a few thousand systems at the most, since going beyond that gets so cookie cutter that it’s not fun anymore. I can certainly see the value of such an approach when dealing with massive environments that are changing all the time. Most companies though this situation doesn’t exist – most companies things are fairly static, you get a new system here and there, you get a new environment maybe once a quarter at the most. Maybe some big project comes along that increases your system count by a large amount for some special purpose.

I have absolutely no problem in maintaining separate config files for each environment and having different config stanzas in the config management tool to push those files out. Not only is this approach simpler (in my view) it gives much more, insight – perhaps is a good word into what is actually happening. I mean if you have a template filled with things that are pulling values dynamically from a half dozen or more different sources you really have no idea what that file really looks like until it lands on the server in question. I like to be able to open the file and look at the settings rather than hunt down the various flags and values that can come from these various sources chef provides.

I’m not building new environments every day, the level of change in general is quite small (as it has been over the past decade at companies I have worked at), I don’t need the level of dynamic abilities that Chef provides because it doesn’t help me that much.

I came up with a new saying a few months ago after dealing with Chef. If it’s not friends with sed, awk and grep then it’s not friends with me. Chef, being very developer-centric uses a lot of JSON to store and manage it’s various configurations. JSON is very much not friendly to sed, awk and grep, and so it frustrates me greatly whenever I have to deal with it.

Because we are moving into a self managed data center environment we needed a way to provision systems. My background is Red Hat/CentOS, Kickstart and Cfengine. We have Ubuntu, <nothing>, and Chef. I came up with a system that for now uses VMware templates (my first ever use of VMware templates) and some custom scripting to integrate with Chef and do other provisioning tasks. It works, it’s not as nice as Kickstart but it works. So speaking of this, and JSON there is a bootstrap process Chef needs to do in order to get itself registered and stuff with the Chef service. This involves creating a bit of JSON that Chef can read. The standard way of Chef bootstrap is a sort of push approach, where there is a management agent that waits for a system to be provisioned, then ssh’s to the system and runs a bunch of stuff. I wanted a pull approach, where the system is provisioned and boots up and configures itself. So I came up with this little bash snippet to construct this JSON file

echo -n "Making first-boot.json ..."
echo -n "{ \"run_list\": [ ">/etc/chef/first-boot.json;
export ROLES=`grep ROLE /root/00-50-* |head -n 1 | sed s'/.*=//'g | sed s'/,/ /'g` &&
for ROLE in $ROLES; do echo -n \"role[${ROLE}]\",;done | sed s'/\,$//'g >>/etc/chef/first-boot.json;
echo -n " ] }" >>/etc/chef/first-boot.json

That /root/00-50-* file is a configuration file named after the MAC address of the VM. This is based on my older kickstart stuff which has been extended to support Chef. It stores things like IP address, Host name, default gateway, for the network, then Chef environment, Chef Role(s), and Chef Organization. It’s a simple text file format, that looks like VARIABLE=value, one VARIABLE per line.

My point with pasting that code is the ugly length I have to go through to simulate valid JSON output using my own regular tool set. Remember I am NOT a programmer!

The scripting works fine(at least so far, built a dozen or so different roles and systems), but it shouldn’t be that complicated.

For those of you more experienced with Vmware templates I noticed there is the ability to customize a template so that Vmware can set the IP address, host name etc of the guest OS. When I saw this I spent a good two hours trying to get it to work, but no matter what I tried Vmware said my configuration was not supported and it would not let me customize. I have read conflicting reports as to whether or not it is possible on Ubuntu. I am running ESX 4.1 with vCenter 5.0. I think if I was running vCenter 4.x it would work fine, but Ubuntu and other “non tier 1″ operating system support for template customization is no longer supported in the 5.0 products. Often times when I see “not supported” especially when something used to work, it means that it might work but don’t ask us for help if it blows up. Maybe coincidence or not but as I said no matter what I did, the customization boxes were greyed out and I could not get vCenter 5.0 to work with Ubuntu.

At the end of the day it doesn’t matter though, I had, what was to me at least a good provisioning process I could adapt from my Kickstart days, a process that works well on both physical as well as virtual machines. Something that leverages the MAC address or the serial number(in the case of physical machines) for unique identification.

With regards on how I used to do things with Cfengine, it was simpler than Chef. Cfengine operates more on trust than Chef. Chef uses public/private keys to authenticate systems, and these keys have to be in the right place in order for a system to get registered. This is good for untrusted networks, like public clouds(ugh). Cfengine works more on trust, where you can (or at least I did) assign network ranges where the IPs are trusted, and a new system could just register itself without any special configuration. The keys would be generated automatically and exchanged between cfengine client and server. I had my cfengine configuration, for the most part dynamic based on the host name of the server. Most of my major Cfengine classes ran a simple grep on the file name that had the host name in it, if the host name matched a particular pattern it was automatically included in the right classes. With Chef life is different, I can’t do that. I have to specifically define which role(s) or recipes a system has up front. Because the system will only download cookbooks that it is specifically configured for using. This isn’t a big deal but is an extra step that I’m not used to having to do.

Sample CFengine class defitition:

ENV_CORPDMZ     = ( ReturnsZero(/bin/egrep -q "^HOSTNAME=corpdmz" /etc/sysconfig/network) )

With Cfengine, prior to implementing the hostname-based approach, adding a new server with Cfengine involved manually editing the master cfengine configuration so that it was aware of the new system that was about to come online. I still had to edit this file on occasion, if there were special configs needed for a server, but for the most part, for like systems, web servers and the like I did not.

Which sort of brings me to the next topic – recruiting talent that can use Chef.  I’ve been managing server systems for about 17 years now, wow has it really been that long.  It’s clear to me after 18 months of chef I lack the knowledge to be able to effectively use the tool (though it hasn’t stopped me from using it at this point), but knowing that, and working with people at my previous company with Chef and seeing the tool present them with a similar level of frustration (if not more), I can see Chef being a real sticking point finding talent that is capable of managing it. My company is actively recruiting senior systems people(well one person) and the candidates that I have spoken with so far, along with candidates I have spoken to in the past, I honestly can think of perhaps one or two people over the years that I know that could handle Chef, and one of them is a full time programmer now (when I met him he was hired to be on my operations team back in 2003).

Well short of the co-worker I have now who does quite a wonderful job in deploying and managing Chef, who wrote the vast majority of Chef stuff at my current company. It’s really well done, but even now that a lot of the hard work was done by him, in a very chef-like way I constantly struggle to add new stuff in, or to change existing things because it’s so dynamic. I see a value for something – where is it coming from? is it from the node? environment ? data bag? attribute? something else?

So I see Chef somewhat like I see Hadoop as far as what skill sets are needed and who can provide them. One of my previous companies was working on migrating towards Hadoop and a big complaint I heard from them about Hadoop (and I have heard it from others since) is finding talent that knows the product. With the likes of Yahoo, Google, and other big companies with very deep pockets and big data aspirations they can afford to pay out the wazoo for Hadoop talent, something small companies just can’t compete with. The number of people qualified to do Hadoop right vs the number of people that can do SQL, well it’s obvious, right.

I see the same with Chef. It’s a powerful tool but it’s just not there yet with regards to usability, I can see it being a very useful tool for the likes of those same kinds of companies who manage very huge fleets of systems and have a very dynamic environment. One such place is HP, whom someone I know is going to work for HP Cloud, because he knows Chef. I assume he is probably pretty good at Chef by now, though the caveat with him is he has a strong Ruby programming background. So it’s no real surprise that he could pick Chef up.

I filed several feature requests and bug reports on the Chef support site about a year ago when I was first interacting with it, though I don’t think much made it through. One thing I’d really like is a good way to do in-line editing of text files. At least at the time the Chef mantra was “find another way to do it”, which a friend of mine says is the same thing Puppet people say. So how do I go about adding an entry to /etc/hosts?

Another thing I’d like to be able to do is bulk file copies from the cookbook and preserve ownership and permissions from the source files(e.g. having a directory tree with various owners/groups/permissions and copying it all at once), I don’t think that is possible still. At the time the Opscode people suggested I use rsync for that.

Another thing I’d like is to be able to host cookbooks internally while using the external service for other things. This is mainly for security purposes I feel more at ease when my core data stays within the confines of my network, on systems under my direct control.

Another thing I’d like to see which I have mentioned to Opscode in one way or another as well is a more abstracted configuration language. I think I called it idiot mode or something. The Ruby syntax they use, while I’m sure it’s great for ruby people really sucks for people like me. I’m fine with a reduced subset of functionality that may be provided by idiot mode, because it’s likely that I won’t use that functionality to begin with(at least not initially). Make the learning curve to actually using the tool less steep.

At one point Opscode was interested in talking to me about a full time position being an advocate for their platform. I just couldn’t go through with it, I just can’t get excited about the platform after all the frustration it has given me. I certainly see the promise and will continue trying,  but I think some fundamental things need to be done to the system in order to make it more usable.

So, in the end, I see Chef as a very powerful tool, a very useful tool for those with the skills that can handle the power it gives you. If I were deploying a new environment today I would certainly NOT use Chef, I would use Cfengine. I don’t want to discourage people from using Chef, it is a good tool, just realize the much higher level of investment you need in order to properly leverage it and try to weigh that against the benefits. For me, the hard things that are made possible by Chef really involve a trivial amount of time. I dare say I have spent FAR more time trying to work with Chef on these hard things (understanding the concepts, code etc) than just flat out doing it by hand the old fashioned way.

You might want to ask – why haven’t I tried Puppet? My answer would be – to-date I haven’t had a reason to. I’ve had a few brief discussions with people who use Puppet over the years(including those who have used Cfengine as well) and asked them why should I use Puppet over Cfengine. For the most part the response was there’s nothing really revolutionary in Puppet so if your happy with Cfengine then stick to it. There are a few things Puppet apparently does better (What they are I don’t remember), but in my talks with people there wasn’t anything — anything that made me want to jump on Puppet. There was things that sounded nice (like Chef has), but not enough return to justify the investment in time to make a migration when, as I mentioned earlier Cfengine does pretty much everything I need it to do.

With Cfengine I could probably train a systems person up on the basics in literally an afternoon. My Cfengine configurations were not complicated. With Chef, well here I am at 18 months and still lost.

3,400 words, I think that’s a record for me for a published blog post. Should get back to sleep now, started writing this at about 3:30AM.

   Send article as PDF to

For the AMD Opteron 6200 and 4200 series (Family 15h) processors, ESX/ESXi 4.1 Update 2 treats each core within a compute unit as an independent core, except while applying licenses. For the purpose of licensing, ESX/ESXi treats each compute unit as a core. For example, a processor with 8 compute units can provide the processor equivalent of 16 cores on ESX/ESXi 4.1 Update 2. However, ESX/ESXi 4.1 Update 2 only requires an 8 core license for each 16-core processor.

I had not heard of that before, so it’s news to me! So not only is the physical cost of the Opteron 6200 cheaper than the 6100, the licensing cost is half as much (per core). AMD’s blog post above shows some pretty impressive results where a pair of quad socket 6200 blades outperforming a pair of quad socket 10-core Intel blades(2 sockets populated per blade) and at the same time the 6200 solution costs half as much (per VM). Though it’s also comparing vSphere 4.1 vs 5.0, since the Opteron 6200 results seem to be the first vSphere 5.0 VMmark results posted. Also the Intel solution has twice the ram as the Opteron but still loses out.

Based on what I see it seems VMmark is more CPU bound than memory(capacity bound), which I suppose I can understand but still in the vast majority of situations the systems are not CPU bound. People tend to load up more on CPUs so they can get more memory capacity. I won’t have real numbers for probably two months but I’m expecting CPU usage on this new cluster I am building to be at least half the amount of memory usage.

The change sounds Oracle-esque in licensing where they have fairly complicated decisions they made to determine how many “Oracle cores” you have on your physical processor.

I am traveling tonight to Atlanta to deploy a new vSphere cluster with Opteron 6100s, I was going to go with vSphere 5 because of the license limits on vSphere 4.1 not supporting 16 core processors. Now I see 4.1 does support it so I have about 48 hours to think about whether or not I want to change my mind. I do like vSphere 5′s inclusion of LLDP support, more vCPUs per VM. Though really even now after I have been looking through what is in vSphere 5 I don’t see anything game changing, nothing remotely, in my opinion like the change to vSphere 4.0 from ESX 3.5.

Weigh the benefits of what’s new in vSphere 5 vs having the ability to have unlimited memory(well, up to 1TB, which for me is unlimited from a practical standpoint) in my hosts for no additional licensing cost…

I’m already licensed for vSphere 5 since we bought it after the deadline of the end of September.

Mad props to AMD for getting VMware to tweak their licensing.

Decisions, decisions..

   Send article as PDF to

I remember reading a while ago how Netgear teamed up with Extreme to re-sell their Black Diamond series of switches.

I didn’t think too much of it till I was on Netgear’s site today so I decided to try to poke around and see if I could find the product(s) that were being resold or OEM’d, and I found them, here is one.

The Netgear 8800

When I saw that it just looked so strange! It’s HP-grey in color, not the usual purple I’m used to seeing. Speaking of HP and purple someone at HP recently speculated to me that the 3PAR arrays will likely stick to being yellow instead of HP-grey because it makes them stand out in the data center.

Tangent comin’ hold onto your butts…

While troubleshooting my home network this morning I think I let some of the smoke out of my HP workstation. Which reminds me of this quote I came across on slashdot years ago

 There is no such thing as a "safe" capacitor! They are filled with SMOKE and that smoke is DEADLY. ALWAYS let the smoke out of the capacitors before attempting to handle them! This should only be done by PROFESSIONALS. Do NOT try this at home.

 Always assume a CAPACITOR is holding a charge. And: Capacitors don't kill people, it's the circuit of which the person is a part that is dangerous...

I thought the networking issue may of been somehow caused by the HP box, so I rebooted it, while it was in the midst of rebooting(middle of POST before the screen came up), I powered it off(by holding down the power button), to reset the network chipset entirely. When I did that I heard a weird clicking sound coming from either the HP box(I think so) or my Cyberpower UPS which was right next to it. Within about 10 seconds I swear a little puff of smoke came out of the HP box(I think), there’s a remote chance it was just dust but I don’t think so. I unplugged the HP box and the clicking stopped. Then I plugged it back in about 30 seconds later, which caused it to turn on automatically, it booted like a champ, no errors, the UPS event log reported nothing. So I don’t know what inside the HP box released the smoke but I guess it was not vital?

Back on topic..

Anyways I poked around in the user manuals and they did a pretty good job of replacing all references of the original product and making it look like a Netgear product through and through (with a couple minor exceptions in diagrams).

I remember about 11 years ago now when I was shopping for a Summit 48 on Ebay for my company(this product wasn’t known for quality at the time though I didn’t know it at the time), I came across some Compaq OEM’d Summit 48s that I think were white in color.

If I was building a bigger network I really would be tempted to opt for this Netgear product if for nothing else to see the expression on people’s faces when I tell them I’m using Netgear, not a brand that comes to most people’s minds when it comes to data center networks! Speaking of data center it looks like Extreme’s 40GbE offerings are leading the market pretty good, I’m so proud of them! Hopefully they can sustain the execution and gain market share. They’ve had some missteps in the past which has knocked them back a few notches(at the time), but they certainly have another opportunity here.

I remember when HP used to OEM/re-sell Foundry Networks chassis switches though I seem to recall HP not making any modifications to the chassis itself(at least according to the pictures on the website, I don’t think it even had an HP logo on the thing). The product at the time was the MG8 which I was entertaining for a data center build out back in 2004/2005. I wasn’t going to buy from HP but was just one of those days that I was poking around and came across it on HP’s site.

Oh and in case your wondering my home network used to be powered by Extreme, I had a trusty Summit 48 for many years, which I eventually upgraded to a Summit 48si (which I still have now). I stopped using it many years ago because I just didn’t have enough ports at home to justify the power usage or more importantly the noise, 1U data center switches are so noisy for home use! I went so far as to replace all of the fans in the 48si (I believe I used Sunon Maglev fans) with quieter ones which reduced the noise by at least half but it was still really loud.

The patented MagLev design is based on magnetic principles and forces that not only propel the fan but also ensure stable rotation over its entire 360 degrees of movement. Utilizing the attraction of the magnetic levitation force, MagLev eliminates the wobbling and shaking problems of traditional motor fans. With this new technology, the MagLev fan propeller is suspended in air during rotation so that the shaft and bearing do not come into direct contact with each other to create friction.

(I dig technology even when it comes to fans!)

The Summit 48 by contrast was 2U and had 80mm fans which spun slower and were quieter. At the moment I have 9 devices wired into my netgear-powered home switching network (one 8-port switch and one 5-port). I used to have a couple Foundry load balancers, and a Cisco switch and a couple other things I think but I recycled them with my Summit 48 years ago, was too lazy to try to re-sell them.

I just saw that picture and was just fascinated by it. It also gives me another opportunity to add more color onto this blog.

   Send article as PDF to

HP Decided to Open source the whole thing, end-to-end

HP intends to fully open source every component of webOS, right down to its Linux core. They aren’t ready to give a public timeframe for exactly when that open sourcing will happen, because there’s some work that has to be done first.

I think it’s wonderful news myself, I did not relish the thought that so many people wanted the likes of Amazon to acquire WebOS which made absolutely no sense to me.

Maintaining an operating system is a lot of work, and I can understand other mobile device makers not wanting to take on the mammoth task of doing so themselves.

Open sourcing it I believe will give it a bright future.

I really was not expecting HP to maintain such.. commitment to the platform. I’m sure if the previous CEO was still around they would not be investing to open source the software, so score a big one for Meg in my book with this decision, good job, this is a great x-mas present for WebOS users and fans alike.

It may be a year or two before another company attempts to release a WebOS product based on this code base but I’m more confident now than I was before that it’ll happen. The OS really is great, and now that it will be pretty much the most technologically advanced fully open mobile platform on the planet (more so than Android I believe) it’ll come back from the depths of the abyss it was in just a few months ago.

I always felt that the likes of the Touchpad, the Veer, and the Pre- were released before they were ready, they were rushed to market. More so on the software front than hardware for the former, more on the hardware for the latter. The Pre3 was going to be released 6 months too late to make any sort of dent in the market.  I think there was too much pressure to deliver products in such a short time frame they didn’t have the time they needed to do it right. With open sourcing of the OS, things will likely move at a slower pace at least initially but the chances of getting it right I believe go way up.

wooohoo!

(I use my Pre3 and one of my two Touchpads every day)

   Send article as PDF to

I was having some talks about this with some co-workers today, it started when buy.com sent me an email from their collections department saying that I owed them money on something that I ordered on August 12th 2011. I got the item a long time ago, and buy.com is one of those places that charges you the same time the item ships. Well I guess this order must’ve slipped through their system somehow (probably along with a bunch of others too). It was an order for a carrying case for a laptop.

I’ve been a happy and loyal customer of buy.com for almost a decade now, my email from them goes back to December 24th 2001. buy.com and outpost.com (now Frys.com) were two of my main shopping sites early on. I still do a bunch of business with buy.com, but haven’t bought from frys.com since they were bought I think.

This was the first such collections email I had received so I thought maybe it was a scam, or a spam or something but the email address I use is not easy to guess and they had all of the right information. So I went to the site and gave them an updated credit card for them to use. You may remember my email system strips out things like links that point to spoofed sites.

The reason the transaction failed is well they weren’t supposed to wait four months before attempting to charge the card. I have been using temporary credit cards generated from Bank of America’s ShopSafe program for many years now, and my standard is to keep the expiry date of the temporary cards to the minimum of two months. So when buy.com’s systems failed and they went back to charge an old order of course it failed since the card was expired.

(Think 4 months is bad? One company I worked for had to literally cut their largest customer a $1M check to sort out an accounting problem(the large customer was and is a publicly traded company so this came up during their quarterly review for earnings) due to their billing system not properly billing customers for something like six months and they weren’t able to go back and charge them too much time had passed)

Anyways my point is Bank of America is not the only bank out there that has this ability I know others do (though can’t think of any off the top of my head I’m sure they exist). What is sort of shocking to me though is over the years as the holidays come and go there are news casts and stuff that try to give tips for how best to protect yourself with online shopping. Usually the tips are good but I’ve never — ever seen anyone mention taking advantage of temporary credit card numbers to better protect your online purchases. I use them even with merchants I trust like buy.com. I think it’s a good habit to be in. I booked my recent hotels and airline tickets with them. Not only have I not heard the media report on this technology but I very rarely come across anyone that even knows such stuff exists (or if they have heard of it, they have never used it).

There’s really only one place that I haven’t been able to use temporary credit card numbers online — that is wingstop. Damn place wants to see the card you used in person when you go to pick your grub up, and I don’t know about you but I haven’t owned a printer in seven years.

These temporary cards can be more labor intensive if your ordering through marketplace sites like buy.com, because the cards only allow one merchant to charge them, so if you have an order with stuff from 5 different merchants you have to split it up into 5 different orders, or use a regular credit card (I of course go ahead and split it up since there is no savings for shipping or anything).

The temporary credit cards also work well for recurring payments. I use one for my co-location bill for this blog. The bill is $100/mo, I think my temporary card for that has a limit of $120/mo, and it allows up to $120/mo each month for as long as I want. It doesn’t allow more than $120/mo though. I can even arbitrarily increase the limit (but I can’t decrease it). I can even arbitrarily extend the expiry date (but I can’t decrease that either). It’s really handy.

When I signed up for a premium subscription to LinkedIn, they sent me an email saying they’ve automatically opted me in to auto renew when my subscription is due next year and that I need to do something special to get out of it. Joke’s on them though, the credit card used to sign up will not only be long expired by then but I have the ability to go into the Shop Safe system and pro-actively delete the credit card whenever I want!

Having a hard time getting someone to stop renewing auto payments? Next time use a temporary credit card!

Now they aren’t perfect. For example there is really no way to tell what charges map to what credit card numbers, and customer service really has little knowledge of this program. Not only that but I think when it comes to fraud, there is no distinction between Shop safe numbers and regular card numbers. If a shop safe number is compromised I DON’T CARE. Chances are it has less than $10 in credit left on it, assuming it is not expired and assuming I haven’t gone in and nuked it anyways! But the bank doesn’t have a way to distinguish it (I don’t think), so if a merchant reports the card # I used was stolen then they flag the account like they otherwise would.

With Shop safe (or a similar program) I’m so much better protected in the online world than I am in the real world. Not that I have much to worry about anyways, it’s not as if I’m liable for fraudulent transactions on my card(I did have a few about a year ago though my card was canceled before I knew what was going on and while the bank said they were going to send me something that disclosed what was bought that never happened, I just signed some paper saying purchases from those merchants was not me).

But I like to be a safe shopper anyways, whenever I can.

So spread the word if you can – use temporary credit card numbers for online shopping for about the most safe shopping experience around!

If you know what other banks offer this capability leave a comment!

Happy Holidays from techopsguys^H

(that makes 100 blog posts for 2011 ! woohoo! I don’t think I’ll come close to the 132 I did last year at this rate)

I started more formally collecting stats on the traffic on the site if your curious check this site out(updates each Sunday). The number one search term for December for my site? Of all things? The HP Touchpad ? I think I have one, maybe two posts on that (plan to have a follow up once HP makes their big decision). OK looks like 4 posts.

I had one crazy IP from Rackspace hosting hitting this site about once every two seconds, 280,000 hits over the span of about 6 weeks, I blocked them at Apache with a message asking them to fix their bot or justify their traffic to me and I’d unblock it but got no reply. They continued for a few more weeks after I blocked them. Really strange!

My data goes back to the middle of August 2011.

   Send article as PDF to

I have told some folks recently that my private rants about EC2 and associated services makes me feel like those folks in 2005-7 screaming about the implosion of the housing market yet for the most part nobody was listening because that’s not what they wanted to hear.

Same goes for EC2.

Anyways, I wanted to mention this, which talks about impending rolling outages across the Amazon infrastructure (within the next week or two).

Oh wait these are not outages, these are “scheduled maintenance events”.

That you can’t opt out of. You can postpone them a bit, but you can’t avoid them entirely, short of getting the hell outta there (which is a project I am working on – finally! Going to Atlanta next week, more than 4 months later than I was originally expecting)

Yeah, good design there. Better design? Take a look at what the folks over at a provider in the UK called UltraSpeed does, it’s clear they are passionate about what they do, and things like 15 minute SLA for restoring a failed server show they take pride in their work(look ma! No hard disks in the servers! Automated off site backups to another country!). Or Terremark – fire in the data center? No problem.

I have little doubt this is in response to critical security flaws which can only be addressed by rebooting the tens or hundreds of thousands of VMs across their infrastructure in a short time before it gets exploited, assuming it’s not being exploited already.

I fully expect that perhaps by the end of this month there will be some security group out there that discloses the vulnerability that Amazon is frantically trying to address now.

   Send article as PDF to

Red hat apparently just released RHEL 6.2, and among the features, is something that looks remarkably similar to UML -

Linux Containers
•    Linux containers provide a flexible approach to application runtime containment on bare-metal without the need to fully virtualize the workload. This release provides application level containers to separate and control the application resource usage policies via cgroup and namespaces. This release introduces basic management of container life-cycle by allowing for creation, editing and deletion of containers via the libvirt API and the virt-manager GUI.
•     Linux Containers provides a means to run applications in a container, a deployment model familiar to UNIX administrators. Also provides container life-cycle management for these containerized applications through a graphical user interface (GUI) and user space utility (libvirt).
•     Linux Containers is in Technology Preview at this time.

Which seems to be basically an attempt at a clone of Solaris containers. Seems like a strange approach for Red Hat to take given the investment in KVM. I struggle to think of a good use case for Linux containers over KVM.

Red hat also has enhanced KVM quite a bit, this update sort of caught my eye

Virtual CPU timeslice sharing for multiprocessor guests is a new feature in Red Hat Enterprise Linux 6.2. Scheduler changes within the kernel now allow for virtual CPUs inside a guest to make more efficient use of the timeslice allocated to the guest, before processor time is yielded back to the host. This change is especially beneficial to large SMP systems that have traditionally experienced guest performance lag due to inherent lock holder preemption  issues. In summary, this new feature eliminates resource consuming system overhead so that a guest can use more of the CPU resources assigned to them much more efficiently.

No informations on specifics as far as what constitutes a “large” system or how many virtual CPUs were provisioned for a given physical CPU etc. But it’s interesting to see, I mean it’s one of those technical details in hypervisors that you just can’t get an indication from by viewing a spec sheet or a manual or something. Such things are rarely talked about in presentations as well. I remember being at a VMware presentation a few years ago where they mentioned they could of enabled 8-way SMP on ESX 3.x, it was apparently an undocumented feature, but chose not to because the scheduler overhead didn’t make it worth while.

Red Hat also integrated the beta of their RHEV 3 platform, I’m hopeful this new platform develops into something that can better compete with vSphere. Though their website is really devoid of information at this point which is unfortunate.

They also make an erroneous claim that RHEV 3 crushes the competition by running more VMs than anyone else and site a SPECvirt benchmark as the proof. While the results are impressive they aren’t really up front with the fact that the hardware more than anything else drove the performance with 80 x 2.4Ghz CPU cores, 2TB of memory and more than 500 spindles. If you look at the results on a more level playing field the performance of RHEV 3 and vSphere is more in line. RHEV still wins, but not by a crushing amount. I really wish these VM benchmarks gave some indication as to how much disk I/O was going on. It is interesting to see all the tuning measures that are disclosed, gives some good information on settings to go investigate maybe they have broader applications than synthetic benchmarking.

Of course performance is only a part of what is needed in a hypervisor, hopefully RHEV 3 will be as functional as it is fast.

There is a Enterprise Hypervisor Comparison released recently by VMGuru.nl, which does a pretty good job at comparing the major hypervisors, though does not include KVM. I’d like to see more of these comparisons from other angles, if you know of more guides let me know.

One thing that stands out a lot is OS support, it’s strange to me how VMware can support so many operating systems but other hypervisors don’t. Is this simply a matter of choice? Or is the VM technology VMware has so much better that it allows them to support the broader number of guest operating systems with little/no effort on their part? Or both ? I mean Hyper-V not supporting Windows NT ? How hard can it be to support that old thing? Nobody other than VMware supporting Solaris ?

I’ve talked off and on about KVM, as I watch and wait for it to mature more. I haven’t used KVM yet myself. I will target RHEV 3, when it is released, to try and see where it stands.

I’m kind of excited. Kind of because breaking up with VMware after 12 years is not going to be easy for me

   Send article as PDF to

Complaint: 500 channels and nothing to watch

Some people even say they get more than 500 channels. Myself I have channels that number up to 800 or 900 but as far as channels I actually have normal content on(whether or not I watch it) I believe is well under 200, there’s tons of PPV, or “radio” channels, or even duplicates(east/west feeds etc). There just isn’t a very good way to figure out what you may like to watch that your not aware of. One issue I have with a DVR is I sometimes come across shows I really like only to find out they were canceled years ago, since I so rarely see the ads for upcoming shows I don’t know what’s out there. But the good thing about a (good) DVR is you can program it in advance. I remember seeing some early advertisements for the show Falling Skies but the show wasn’t due to air for months at the time(guide data goes out only two weeks), so I set a wish list in my Tivo to record any shows in HD with that name, and right on schedule there it was when the time came.

Complaint: I want ala carte access

A major request seems to be people want to subscribe to only the 10 or 20 channels (maybe less) that have the content they want to watch, pay a tiny amount per channel and cut their bill by a large amount. What people don’t seem to realize is this would (if widely deployed) cause probably at least 50-60% of the channels to go away entirely (the vast majority of which are watched by some niche of people including geeks/nerds) and the net result may be instead of having 10 channels of content they want to watch they’re down to two because the rest got canceled. Subsidizing is an important part of making these channels available.

There is on demand from some cable (and maybe satellite too) companies. I don’t know what the state of that is since I have Tivo which is not compatible with that. Tivo in itself is a form of on demand, one that is compatible with a wide range of cable, OTA and in some cases satellite providers. Though most people don’t want to pay for that either.

A good example of this for me at least was the cancellation of one of the only pod casts/web casts/whatever you call them called Cranky Geeks, due to lack of advertiser interest, Cranky Geeks wasn’t subsidized much (if at all), and it died (had a good run though, I didn’t come across it until the last year or so of operation). I really enjoyed it even if I rarely learned anything from it, it was great to watch the points of view and conversations they had.

Complaint: I want real on demand access

Some people, myself included dream of this, but it’ll likely never happen for reasons that have nothing to do with TV as a product or technology. The same applies for books or music too. It is unfortunate as this would be a holy grail.

One post goes to an extreme

Here’s how it should work. I pay a content provider a subscription for a show. For instance, $0.75 for a season of House without ads, or $0.05 for a season of House with ads. $1.50 for a season of “The Daily Show” without ads.

If that sort of model gets adopted TV will go dark overnight.

Complaint: costs too much

You still have the option of going with over the air broadcasts, I recently bought a high end HD TV antenna for someone as an xmas present so they could get HD signals since they don’t have HD cable service. I don’t think I’ve used OTA since 1989(TV or Radio), I’ll see how well it works these days. All of this content costs money which leads me to..

Complaint: too much advertising

You were just complaining that you are paying too much and now your saying there’s too many ads too ? You either need to pay more, or get more ads or go somewhere else for your (hopefully legal but probably won’t be) content.

There is also a new invention called a DVR, I think TiVo and at one point ReplayTV had them. Tivo has had it’s first increase in subscribers in many years recently, which is nice to see as a very long time Tivo subscriber (3 Tivos all with lifetime subscriptions, gave one to my sister earlier this year). My first Tivo Series 1 is still used every day and I bought it about 10 years ago (April 4th 2001 in fact just looked at the receipt – replaced the HDs to get more capacity about 6 years ago). To me at least it’s kind of sad to see what passes as a DVR these days from the cable/satellite companies, but I suppose they are sometimes better than nothing.

Complaint: give me more educational programing less mindless sitcoms

If there were more people that were willing to watch such content such content would appear, since there is not, the market goes to where the money is. I saw one person ask for college lectures on TV, these do exist although are probably pretty rare. I skimmed across some when I was at my mother’s house a couple of weeks ago in Orange County. There is a community college there called SaddleBack Community College which airs a lot of stuff on cable TV, they have at least one dedicated channel. I attended this school myself for a year or so back when I was experimenting with the idea of going to college, some of the classes I took I had to watch the TV to get the lectures, then study at home and go in and take tests once in a while. It didn’t work for me any more than regular school works for me, I could study, take notes, and highlight passages in a book for an hour and at the end not remember a single thing of what I just did (even though the notes and highlights were good) because my mind simply wasn’t in it.

So instead of blaming TV you should be blaming the audience since TV is giving the audience what most people want.

Guess what – Fear factor is coming back.. Reality shows are cheap to produce and attract a good number of viewers. My favorite reality shows are the car repo shows on TruTV – it’s amazing to me to see the lengths some folks go through to prevent their property from being re-possessed by the bank. Those repo guys in New Jersey have some serious balls. Looks like you can watch the shows online, but when I go to the link nothing happens – must not be compatible with Linux.

My complaints

I share the same above complaints but they are really minor for me, most of them are addressed by Tivo, and having had Tivo so long it’s a natural thing at this point. I do wish I could find a better way to find things on TV. I have Tivo suggestions, and I can rate shows with thumbs up/down on my remote which is a start but I’m sure there is more out there, e.g. on the premium channels my usage of them has been very very low for years now, I’m sure there is some good stuff here or there but am not exposed to it. I see many people sing the praises of HBO though I can’t remember the last time I was tuned to HBO(not sure why I’ve kept them around). Tivo did go through a phase with me at one point where it was confident I was a fan of kung fu stuff and kept recording tons of kung fu over and over. I’ve never been a fan o f kung fu and had never recorded anything, took a few weeks of convincing, but Tivo eventually stopped recording kung fu.

I don’t like how badly digital tv degrades when there are signal issues, the audio cutting out is the most annoying. Analog TV degraded so much more gracefully!

I would like a next generation of cable card that was more functional, something that had more cable box-like functionality so I could do things like get on demand, or order PPV from Tivo. Not critical though.

More remote access would be nice, I have a couple of Slingbox solo’s that I haven’t had hooked up in 4-5 years but plan to hook one up this weekend for my trip to Atlanta next week so I can watch my local NFL games while I’m there (assuming there is time). Slingbox had an annoying firmware update a few years back which did some stupid things, fortunately I never accepted the update so I can still use my Slingbox in Linux. Would be great if I could use it from my WebOS devices but that’ll never happen. So even remote access is addressed to some degree.

I’d like to see more creative use of the additional resolution provided by HD on TV. CNBC HD+ does a great job at this, very cool. I’ve seen other channels like NBC do creative things during big political elections. Maybe it wouldn’t work for most other channels but for news (and maybe sports) it would be cool. I think NFL Redzone is sort of like this as well. Though I’ve only seen NFL Redzone at a bar once in Arizona years ago(and seem to remember having multiple games on at the same time on a single screen). I don’t watch it normally since I like to watch the full game.

Last Thoughts

A good DVR goes a long way to me. I just checked the number of season passes and it’s more than 70 at this point (all time high for me). I think some shows are canceled and I just don’t know it yet, most of the season passes transferred flawlessly when I moved to CA (I thought I’d lose them all), I did lose those that were tied to local TV channels (since those channels don’t exist here – the passes stayed but the shows didn’t get recorded), but the cable channel ones all stuck around.

A lot of what I watch is news (CNBC mainly, find it entertaining, I’m not an investor), and I watch some NFL too(speaking of sports I find it funny that so many people feel they have to watch it live, as if the results of the game would change if they watched it time delayed, I don’t start watching a NFL game until at least and hour after it starts), and a bunch of other things some of which I could probably get online, but the Tivo gets it for me already and has been very reliable so haven’t had a reason to change.

Speaking of DVRs, I saw an interview with the president of CBS earlier this morning on CNBC where he actually said DVRs are our friends. Which I thought was shocking, given how easy it is to skip ads and stuff. His reasoning was it boosts viewer ship of shows by a big margin (on big shows the numbers are in the millions). I guess this combined with product placement advertising and stuff offsets the ability to time shift.

I’ve covered it many times before but the current design of the internet doesn’t have the capacity to replace broadcast and satellite tv, it’s just not efficient enough by a very wide margin. If IPv6 and large scale multicast come along, assuming it works maybe it would work, I’m not holding my breath though.

I’ll admit the price tag of Tivo isn’t cheap, it wasn’t an easy decision way back when to spend what I think was around $950 for a DVR,and my series 3 unit which is my newest but still four years old was around $1,300 (just looked at the receipt – March 2007 included storage upgrade). But they have paid for themselves many times over during the years, especially given how long they have lasted and the lifetime subscriptions (which have gone up over the years, originally for me it was $199, next Tivo was $299, most recent Tivo Series 3 was $399, now it’s $499). I remember reading at one point that “lifetime” technically meant something like 3 years with Tivo (in the fine print – this was ~7-8 years ago not sure about today), fortunately Tivo hasn’t declared my oldest Series 1 units obsolete and stopped serving them, probably because there’s still a fair number of those boxes out there and if they stopped serving them they would take a decent hit to their subscriber numbers and they don’t want to take that hit. They did cut off UK users a while back when they pulled out of the market.

My Tivos probably wouldn’t lasted as long as they have if I didn’t protect all of my electronic equipment with UPSs which I believe goes a long ways towards extending their life. It almost pains me to see someone with fancy electronic gear just plugged into a cheap surge protector. UPSs at home are so incredibly rare outside of the tech community.

Tivo isn’t exactly rolling in spare dollar bills so it’s hard to say they are ripping me off for this stuff, it costs a lot of money to do good R&D, and they don’t have large economies of scale to drive the costs down so the cost remains higher than it otherwise would be. People have been writing the obituary for Tivo for what seems like a decade now and yet it keeps on kicking (fortunately for folks like me!).

With the Tivo upgrade program I can get the lifetime service for $399 again and retain my existing Tivo subscriptions (it doesn’t transfer it just gives a discount which is nice), I have contemplated over the years whether or not to get the next generation of Tivo but so far haven’t had a big reason to do so. A lot of the new functionality revolves around internet streaming (some of which I have on my current Series 3 but don’t really use it much – haven’t come across much I’m interested in), and Apple iPad and iPhone integration (I have no Apple products so that doesn’t help me either).

I don’t think TV is broken, the basic model is outdated to some extent, but bolt ons like on demand, and DVRs are helping extend the useful life of the concept, the lack of patience of the younger generations will result in the continued slow erosion in market share though.

The lack of patience reminds me of this from Star Trek.

[..]
STUBBS: Nobody will say anything at all, Wesley. We will not even be mentioned. I could live with failure. Well, maybe not. But never even to try. To miss your one chance at bat. Do you know baseball?
WESLEY: Yes, my father taught it to me when I was young.
STUBBS: Once, centuries ago, it was the beloved national pastime of the Americas, Wesley. Abandoned by a society that prized fast food and faster games. Lost to impatience.
[..]

   Send article as PDF to