For me it started back in 2006 when I went to work for my first social media startup, a company that very quickly seemed to lose it's way and just wanted to try to capitalize on Facebook some how. Myself I did not(and still don't) care about social media, one of the nice things about being on the operations/IT/internet side of the company is it doesn't really matter what the company's vision is or what they do, my job stays pretty much the same. Optimize things, monitor things, make things faster etc. All of those sorts of tasks improve the user experience no matter what and I don't need to get involved in the innards of company strategy or whatever. I mistakenly joined another social media startup a few years later and that place was just a disaster any way you slice it. No more social media companies for me! Tired of the "I wanna be facebook too!" crowd.

Anyways, back on topic. The forthcoming Facebook IPO. The most anticipated IPO in the past decade I believe anyways. Obviously tons of hype around it but I learned a couple interesting things, yesterday I think, that made me chuckle. This information comes from analysts on CNBC - I don't care enough about Facebook to research the IPO myself it's a waste of time.

Facebook has a big problem, and the problem is mobile. There hasn't been any companies that have been able to monetize mobile(weird thinking I worked at a mobile payments company going back to 2003 that was later acquired by AMDOCS which is a huge billing provider for the carriers) in the same way that companies have been able to monetize the traditional PC-based web browsing platform with advertising. There have been companies like Apple that makes tons of money off their mobile stuff but that's a different and somewhat unique model. The point is advertising. Whether it's Google, or Pandora, Facebook, and I'm confident Twitter is in the same boat. Nobody is making profits on mobile advertising - despite all the hype and efforts. I guess the screen is too small.

So expanding on that a bit, this analyst said yesterday that outside of the U.S. and parts of Europe the bulk of the populations using Facebook use it almost exclusively on mobile - so there's no real revenue for Facebook from them at this time.

Add to that apparently Facebook has written China off as a growth market for some specific reason(don't recall what). Which seems contrary to the recent trend where companies are falling head over heels to try to get into China, giving up their intellectual property to the Chinese government(why..why?!) to get into that market.

So that leaves the U.S. and a few other developed markets that are still, for the most part, using their computers to interact with Facebook.

So Facebook is in a race - to be the first company to monetize mobile before their lucrative subscriber base that they have in these few developed markets shifts away from the easy-to-advertise-on computer platform.

Not only that but there's another challenge that faces them as well. Employee retention. Myself of course would never work for Facebook, I've talked to several people that have interviewed there, and a couple that have worked there and I've never really heard anything positive come out of anyone about the company.

Basically it seems like the only thing holding it together is the impending IPO. In fact at one point I believe it was reported that Zuckerberg delayed the IPO in order to get employees to re-focus on the company and software and not get side tracked by the IPO.

So why IPO now? One big reason seems to be taxes, of all things. With many tax rates currently scheduled to go up on Jan 1, 2012 - Facebook wants to IPO now, with the employee lock up preventing anyone from selling shares for six months - that gets you pretty close to the New Year, and the potential new taxes.

The IPO is also expected to trigger a housing boom in and around Palo Alto, CA. I remember seeing a report about a year ago that mentioned many people in the area wanted to sell their houses but were holding off for the IPO - as a result the housing market(at least at the time, not sure what the state is now) was very tight with only a few dozen properties on the market out of tens of thousands.

There was even a California politician or two earlier in the year that said the state's finances weren't in as bad of shape as some people were making out because they weren't taking into account the effect of the Facebook IPO. Of course recently it was announced that things were in fact, much worse than some had previously communicated.

I'm not saying the hype won't drive the stock really high on opening day - wouldn't surprise me if it went to $90 or $100 or more. It seems like the IPO road show that Facebook took, in their case it felt like a formality more than anything else. I just saw someone mention that in Asia the IPO is 25X oversubscribed.

One stock person I saw recently mentioned her company has received more requests about the Facebook IPO than any other IPO in the past 20 years.

Maybe they can pull mobile off before it's too late, I'm not holding my breath though.

I really didn't participate in the original dot com bubble, I worked at a dot com for about 3 months in the summer of 2000 but that was about it. So this comparison may not be accurate but the hype around this IPO really reminds me of that time, I'm not sure how many of the original dot com companies you would have to combine to reach a market cap of $100B,  hopefully it's 100 at least. But it's sort of like a mini dot com bubble all contained within one company. With so many other wanna be hopefuls in the wings not able to get any momentum to capitalize on it beyond their initial VC investments. The two social media companies I worked for combined got around  I want to say $90M in funding alone.

Another point along these lines, is the esteemed CEO of Facebook seems to be on a social mission and cares more about the mission than the money. That reminds me so much of the dot com days, it's just another way of saying we want even more traffic, more web site hits! Sure it's easy to not care much about the money now because people have bought the hype hook line and sinker and are just throwing money at it. Obviously it won't last though

Myself of course, will not buy any Facebook stock - or any other stock. I'm not an investor, or trader or whatever.

Yes - it seems they are going to have fixed instance sizes and no apparent support for resource pools. This is especially depressing from someone like HP who has technology like thin provisioning, and partnerships with the likes of all of the major hypervisor players.

Is the software technology at the hypervisor just not there yet to provide such a service? vSphere 5 for example supports 1600 resource pools per cluster. I don't like the licensing model of 5, so I built my latest cluster on 4.1 - which supports 512 resource pools per cluster. Not a whole lot in either case but then again cluster sizes are fairly limited anyways.

There's no doubt that gigabyte to gigabyte that DAS is cheaper than something like a 3PAR V800. But with fixed allocation sizes from the likes of Amazon - it's not uncommon to have disk utilization rates hovering in low single digits. I've seen it at two different companies - and guess what - everyone else on the teams (all of whom have had more Amazon experience than me) was just as not surprised as I was.

So you take this cheap DAS and you apply a 4 or 5% utilization rate to it - and all of a sudden it's not so cheap anymore is it ? Why is utilization so low ? Well in Amazon (since I haven't use HP's cloud), it's primarily low because that DAS is not protected, if the server goes down or the VM dies the storage is gone. So people use other methods to protect their more important data. You can have the OS and log files and stuff on there no big deal if that goes away - but again - your talking about maybe 3-5GB of data (which is typical for me at least). Then the rest of the disk goes unused.

Go to the most inefficient storage company in the world and and even they will drool at the prospects of replacing storage that your only getting 5% utilization out of! Because really even the worst efficiency is maybe 20% on older systems w/o thin provisioning.

Even if the storage IS fully protected - the fixed allocation units are still way out of whack and they can't be shared! I may need a decent amount of CPU horsepower and/or  (more likely) memory to run a bloated application but I don't need several hundred gigabytes of storage attached to each system when 20GB would be MORE than enough(my average OS+App installation comes in at under 3GB and that's with a lot of stuff installed)! I'd rather take that several hundred gigabytes both in terms of raw space and IOPS and give them to database servers or something like that(in theory at least, the underlying storage in this case is poor so I wouldn't want to use it for that anyways).

This is what 3PAR was built to solve - drive (way)utilization up, while simultaneously providing the high availability and software features of a modern storage platform. Others do the same too of course with various degrees of efficiency.

So that's storage - next take CPU. The industry average pre-virtualization was in the sub 20% utilization range - my own personal experience says it's in the sub 10% range for the most part. There was a quote from a government official a couple years back that talked about how their data centers are averaging about 7% utilization. I've done a few virtualization projects over the years and my experience shows me that even after systems have been virtualized the vmware hosts themselves are at low utilization from a CPU perspective.

Two projects in particular that I documented while I was at a company a few years ago while back - the most extreme perhaps being roughly 120VMs on 5 servers, four of them being HP DL585 G1s - which were released in 2005. They had 64GB of memory on them but they were old boxes. I calculated that the newer Opteron 6100 when it was released had literally 12 times the CPU power(according to SPEC numbers at least) of the Opteron 880s that we had at the time. Anyways, even with these really old servers the cluster averaged under 40% CPU - with peaks to maybe 50 or 60%. Memory usage was pretty constant at around 70-75%. Imagine translating that workload on those ancient servers onto something more modern and you'd likely see CPU usage rates drop to single digits while memory usage remains constant.

I have no doubt that the likes of HP and Amazon are building their cloud to specifically not oversubscribe - to assume that people will utilize all of the CPU allocated to them as well as memory and disk space. So they have fixed building blocks to deal with and they carve them up accordingly.

The major fault with the design of course is the vast majority of workloads do not fit in such building blocks and will never come close to utilizing all of the resources that are provided - thus wasting an enormous amount of resources in the environment. What's Amazon's solution to this ? Build your apps to better utilize what they provide. basically work around their limitations. Which, naturally most people don't do so resources end up being wasted on a massive scale.

I've worked for really nothing but software development companies for almost 10 years now and I have never really seen even one company or group or developer ever EVER design/build for the hardware. I have been part of teams that have tried to benchmark applications and buy the right sized hardware but it really never works out in the end because a simple software change can throw all those benchmarks and testing out the window overnight(not to mention how traditionally difficult it is to replicate real traffic in a test environment - I've yet to see it done right myself for any even moderately complex application). The far easier solution to take is of course, resource pools, and variably allocated resources.

Similarly this model, along with the per-VM licensing model of so many different products out there go against the trend that has allowed us to have more VM sprawl I guess. Instead of running a single server  with a half dozen different apps it's become a good practice to split those apps up. This fixed allocation unit of the cloud discourages such behavior by dramatically increasing the cost of doing it. You still incur additional costs by doing it on your own gear - memory overhead for multiple copies of the OS (assuming that memory de-dupe doesn't work -which for me on Linux it doesn't), or disk overhead (assuming your array doesn't de-dupe -which 3PAR doesn't - but the overhead is so trivial here that it is a rounding error). But those incremental costs pale in comparison to massive increases in cost in the cloud, because again of those fixed allocation units.

I have seen no mention of it yet, but I hope HP has at least integrated the ability to do live migration of VMs between servers. The hypervisor they are using supports it of course, I haven't seen any details from people using the service as to how it operates yet.

I can certainly see a need for cheap VMs on throwaway hardware. I see an even bigger need, for the more traditional customers(that make up the vast, vast majority of the market) to have this model of resource pools instead. If HP were to provide both services - and a unified management UI that really would be pretty nice to see.

The concept is not complicated, and is so obvious it dumbfounds me why more folks aren't doing it (only thought is perhaps the technology these folks are using isn't capable) - IaaS won't be worth while to use in my opinion until we have that sort of system in place.

HP is obviously in a good position when it comes to providing 3PAR technology as a cloud since they own the thing their support costs would be a fraction of what their customers pay and they would be able to consume unlimited software for nothing. Software typically makes up at least half the cost of a 3PAR system(the SPC-1 results and costs of course only show the bare minimum software required). Their hardware costs would be significantly less as well since they would not need much(any?) margin on it.

I remember SAVVIS a few years ago wanting to charge me ~$200,000/year for 20TB usable on 10k RPM storage on a 3PAR array, when I could of bought 20TB usable on 15k RPM storage on a 3PAR array(+ hosting costs) for less than one year's costs at SAVVIS. I heard similar stories from 3PAR folks where customers would go out to the cloud to get pricing thinking it might be cheaper than doing it in house but always came back being able to show massive cost savings by keeping things in house.

They are also in a good position as a large server manufacturer to get amazing discounts on all of their stuff and again of course don't have to make as much margin for these purposes (I imagine at least). Of course it's a double edged sword pissing off current and potential customers that may use your equipment to try to compete in that same space.

I have hope still, that given HP's strong presence in the enterprise and in house technology and technology partners that they will, at some point offer an enterprise grade cloud, something where I can allocate a set amount of CPU, memory, maybe even give me access to a 3PAR array using their Virtual Domain software, and then provision whatever I want within those resources - and billing would be based on some sort of combination of a fixed price for base services and variable price based on actual utilization (bill for what you use rather than what you provision), with perhaps some minimum usage thresholds (because someone has to buy the infrastructure to run the thing). So say I want a resource pool with 1TB of ram and 500Ghz of CPU. Maybe I am forced to pay for 200GB of ram and 50Ghz of CPU as a baseline, then anything above that is measured and billed accordingly.

Don't let me down HP.

The new products are

• 48 port 10Gbase-T module for the Black Diamond X-series - for up to 384 x 10GbaseT ports in a 14U chassis - note this is of course half the density you can achieve using the 40GbE modules and break out cables, there's only so many plugs you can put in 14U!
• Summit X670V-48t (I assume that's what it'll be called) - a 48-port 10GbaseT switch with 40GbE uplinks (similar to the Arista 7100 - the only 48-port 10GbaseT switch I'm personally aware of - just with faster uplinks and I'm sure there will be stacking support for those that like to stack)

From this article it's claimed a list price of about $25k for the 1U switch which is a good price - about the same price as the existing 24-port X650 10GbaseT product. Also in line with the current generation X670V-48x which is a 48-port SFP+ product, so little to no premium for the copper which is nice to see! (note there is a lower cost X670 (non "V") that does not have 40GbE ability available for about half the cost of the "V" model)

Much of the hype seems to be around the new Intel 10Gbase-T controller that is coming out with the latest CPUs from them.

With the Intel Ethernet Controller X540, Intel is delivering on its commitment to drive down the costs of 10GbE. We’ve ditched two-chip 10GBASE-T designs of the past in favor of integrating the media access controller (MAC) and physical layer (PHY) controller into a single chip. The result is a dual-port 10GBASE-T controller that’s not only cost-effective, but also energy-efficient and small enough to be included on mainstream server motherboards. Several server OEMs are already lined up to offer Intel Ethernet Controller X540-based LOM connections for their Intel Xeon processor E5-2600 product family-based servers.

With Broadcom also having recently announced (and shipping too perhaps?) their own next generation 10GbaseT chips, built for LOM (among other things), which apparently can push power utilization down to under 2W per port, using a 10 meter mode (perhaps?), 10m is plenty long enough for most connections of course! Given that Broadcom also has a quad port version of this chipset, could they be the ones powering the newest boxes from Oracle ?

Will Broadcom be able to keep their strong hold on the LOM market (really can't remember the last time I came across Intel NICs on motherboards outside of maybe Supermicro or something)?

So the question remains - when will the rest of the network industry jump on board - after having been burned somewhat in the past by the first iterations of 10GbaseT.

Anyways I was reading this article on slashdot just now about Skype on Windows phone 7. The most immediate complaint was the design of WP7 prevents skype from receiving calls while in the background because with few exceptions like streaming media and stuff any background app is suspended. There is no multi tasking on WP7? As some others I have seen notice - I haven't seen a WP7 phone on anyone yet, so haven't seen the platform in action. Back when what was Palm was gutted last year and the hardware divisions shut down many people were saying how WP7 was a good platform to go to from WebOS, especially the 7.5 release which was pretty new at the time.

I don't multi task too much on my phone or tablets, but it's certainly nice to have the option there. WebOS has a nice messaging interface with full skype integration so skype can run completely in the background. I don't use it in this way mainly because the company I'm at uses Skype as a sort of full on chat client, so the device would be hammered by people talking (to other people) in group chats which is really distracting. Add to that the audible notifications for messaging on WebOS applies to all protocols, so I use a very loud panic alarm for SMS messages for my on call stuff, and having that sound off every couple of seconds when a skype discussion is going is not workable! So I keep it off unless I specifically need it. 99.9% of my skype activity is work related. Otherwise I wouldn't even use the thing. Multi tasking has been one of the biggest selling points of WebOS since it was released several years ago, really seeming to be the first platform to support it (why it took even that long sort of baffles me).

So no multi tasking, and apparently no major upgrades coming either - I've come across a few articles like this one that say it is very unlikely that WP7 users will be able to upgrade to Windows 8/WP8. Though lack of mobile phone upgrades seems pretty common, Android in particular has had some investigations done to illustrate the varying degrees when or if the various handsets get upgrades. WebOS was in the same boat here, with the original phones not getting past version 1.4.5 or something, the next generation of phones not getting past 2.x, and only the Touchpad (with a mostly incompatible UI for phones apparently) having 3.x. For me, I don't see anything in WebOS 3.x that I would need on my WebOS 2.x devices, and I remember when I was on WebOS 1.x I didn't see anything in 2.x that made me really want to upgrade, the phone worked pretty well as it was. iOS seems to shine the best in this case providing longer term updates for what (has got to be) is a very mature OS at this point.

But for a company that has as much resources as Microsoft, especially given the fact that they seem to be maintaining tighter control over the hardware the phones run on, it's really unfortunate that they may not be willing/able to provide the major update to WP8.

Then there was the apparent ban Microsoft put on all players, preventing them from releasing multi core phones in order to give Nokia time to make one themselves, instead of giving even more resources to making sure they could succeed they held the other players back, which not only hurts all of their partners (minus Nokia, or not?) but of course hurt the platform as a whole.

I'm stocked up on WebOS devices to last me a while on a GSM network. So I don't have to think about what I may upgrade to in the future, I suspect my phones might outlive the network technologies they use.

To come back to the original topic - lack of multi tasking - specifically the inability for Skype to operate in the background is really sad. Perhaps the only thing worse is it took this long for Skype to show up on the platform in the first place. Even the zombie'd WebOS has had Skype for almost a year on the Touchpad, and if you happened to have a Verizon Pre2 phone at the time, Skype for that was released just over a year ago(again with full background support). I would of thought given Microsoft bought Skype about a year ago that they would of/ could of had a release for WP7 within a very short period of time(30 days?). But at least it's here for the 8 people that use the phone, even if the version is crippled by the design of the OS. Even Linux has had Skype (which I use daily) for longer. There have been some big bugs in Skype on WebOS - most of them I think related to video/audio, doesn't really impact me since most of my skype usage is for text chat.

While I'm here chatting about mobile I find it really funny, and ironic that apparently Microsoft makes more money off of Android than it does it's own platform(estimated to be five times more last year), and Google apparently makes four times more money off of iOS than it's own platform.

While there are no new plans for WebOS hardware at this point - it wouldn't surprise me at all if people inside HP were working to make the new ARM-based WP8 tablets hackable in some way to get a future version of WebOS on them, even though MS is going to do everything they can to prevent that from happening.

Someone commented on here in response to my HP SPC-2 results pointing out that the new Oracle 7240 ZFS system has some new SPC-2 results that are very impressive, and today I stumble upon an article from our friends at The Register which talks about a similar 7240 system being tested in the SpecSFS benchmark with equally impressive results.

The main thing missing to me with the NFS results is the inability to provide them over a single file system(not just a global name space as NetApp tries to advertise but truly a single file system), oh, and of course the disclosure of costs with the test.

This 7240 system must be really new, when I went to investigate it recently the product detail pages on Oracle's own site were returning 404s, but now they work.

I'll come right out and say it - I've always been a bit leery of the ZFS offerings for a true high availability solution, I wrote a bit about this topic a while ago. Though that article focused mainly on people deploying ZFS on cheap crap hardware because they think they can make an equivalent enterprise offering by slapping some software on top of it.

I'm also a Nexenta customer for a very small installation (NAS only, back end is 3PAR). I know Nexenta and Oracle ZFS are worlds apart but at least I am getting some sort of ZFS exposure. ZFS has a lot of really nice concepts, it's just a matter of how well it works in practice.

For example I was kind of shocked to learn that if a ZFS file system gets full you can't delete files off of it. I saw one post of a person saying they couldn't even mount the file system because it was full. Recently I noticed on one of my Nexenta volumes a process that kicks in when a volume gets 50% full. They create a quota'd file system on the volume of 100MB in size, so that when/if the file system is full you can somehow remove this data and get access to your file system again. Quite a hack.

I've seen another thread or two about existing Sun ZFS customers who have gotten very frustrated with the lack of support Oracle has given them since Oracle took the helm.

ANYWAYS, back to the topic of exploiting x86-64 architecture. Look at this -

ZFS Storage array base specifications

Clearly Oracle is embracing the processing and memory power that is available to them and I have to give them mad props for that - I wish other companies did the same, the customer would be so much better off.

They do it also by keeping the costs low (relative to the competition anyways), which is equally impressive. Oracle is a company of course that probably likes to drive margins more than most any other company out there, so it is nice to see them doing this.

My main question is - what of Pillar ? What kind of work is being done there? I haven't noticed anything since Pillar went home to the Larry E mothership. Is it just dieing on the vine? Are these ZFS systems still not suitable for certain situations which Pillar is better at supporting?

Anyways, I can't believe I'm writing about an Oracle solution twice in the same week but these are both nice things to see come out of one of the bigger players out there.

One of their claims to fame was the ability to "cluster" their load balancers so that you could add more boxes on the fly and it would just go faster, instead of having to rip and replace, or add more boxes and do funky things with DNS load balancing in trying to balance between multiple groups of load balancers

Crescendo's Scale out design - too bad the company didn't last long enough to see anyone leverage a 24-month expansion

Another company, A10 Networks (who is still around, though I think Brocade and F5 are trying to make them go away), whom introduced similar technology about a year ago called virtual chassis (details are light on their site). There may be other companies that have similar things too - they all seem gunning for the F5 VIPRION, which is a monster system, they took a chassis approach and support up to 4 blades of processing power. Then they do load balancing of the blades themselves to distribute the load. I have a long history in F5 products and know them pretty well, going back to their 4.x code base which was housed in (among other things) generic 4U servers with BSDI and generic motherboards.

I believe Zeus does it as well, I have used Zeus but have not gone beyond a 2 node cluster. I forgot, Riverbed bought them and changed the name to Stingray. I think Zeus sounds cooler.

With Crescendo the way they implemented their cluster was quite basic, it was very similar to how other load balancing companies improved their throughput for streaming media applications - some form of direct response from the server to the client, instead of having the response go back through the load balancer a second time. Here is a page from a long time ago on some reasons why you may not want to do this. I'm not sure how A10 or Zeus do it.

I am a Citrix customer now, having heard some good things about them over the years, but never having tried the product. I found it curious why the likes of Amazon and Google gobble up Netscaler appliances like M&Ms when for everything else they seem to go out of their way to build things themselves. I know Facebook is a big user of the F5 VIPRION system as well.

You'd think (or at least I think) companies like this - if they could leverage some sort of open source product and augment it with their own developer resources they would - I'm sure they've tried - and maybe they are using such products in certain areas. My information about who is using what could be out of date. I've used haproxy(briefly), nginx(more) at least for load balancers and wasn't happy with either product. Give me a real load balancer please! Zeus seems to be a pretty nice platform - and open enough that you can run it on regular server hardware, rather than being forced into buying fixed appliances.

Anyways, I had a ticket open with Citrix today about a particular TLS issue regarding SSL re-negotiation, after a co-worker brought it to my attention that our system was reported as vunerable by her browser / plugins. During my research I came across this excellent site which shows a ton of useful info about a particular SSL site.

I asked Citrix how I could resolve the issues the site was reporting and they said the only way to do it was to upgrade to the latest major release of code (10.x). I don't plan to do that, resolving this particular issue doesn't seem like a big deal (though would be nice - not worth the risk of using this latest code so early after it's release for this one reason alone). Add to that our site is fronted by Akamai  (which actually posted poorer results on the SSL check than our own load balancers). We even had a "security scan" run against our servers for PCI compliance and it didn't pick up anything related to SSL.

Anyways, back on topic. I was browsing through the release notes for the 10.x code branch and saw that Netscaler now supports clustering as well

You can now create a cluster of nCore NetScaler appliances and make them work together as a single system image. The traffic is distributed among the cluster nodes to provide high availability, high throughput, and scalability. A NetScaler cluster can include as few as 2 or as many as 32 NetScaler nCore hardware or virtual appliances.

With their top end load balancers tapping out at 50Gbps, that comes to 1.6Tbps with 32 appliances. Of course you won't reach top throughput depending on your traffic patterns so taking off 600Gbps seems reasonable, still 1Tbps of throughput! I really can't imagine what kind of service could use that sort of throughput at one physical site.

It seems, at least compared to the Crescendo model the Citrix model is a lot more like a traditional cluster, probably a lot more like a VIPRION design -

The NetScaler cluster uses Equal Cost Multiple Path (ECMP), Linksets (LS), or Cluster Link Aggregation Group (CLAG) traffic distribution mechanisms to determine the node that receives the traffic (the flow receiver) from the external connecting device. Each of these mechanisms uses a different algorithm to determine the flow receiver.

Citrix Netscaler Traffic Flow

The flow reminds me a lot of the 3PAR cluster design actually.

My Thoughts on Netscaler

My experience so far with the Netscalers is mixed, some things I really like such as an integrated mature SSL VPN (note I said mature! well at least for windows - nothing for Linux and their Mac client is buggy and incomplete), application aware MySQL and DNS load balancing, and a true 64-bit multithreaded, shared memory design. I also really like their capacity on demand offering as well. These boxes are always CPU bound, so to have the option to buy a technically lower end box with the same exact CPU setup as a higher end box (that is rated for 2x the throughput) is really nice. It means I can turn on more of those CPU heavy features without having to fork over the cash for a bigger box.

Citrix nCore

While for the most part, at least last I checked - F5 was still operating on 32-bit TMOS (on top of 64-bit Linux kernels) leveraging a multi process design instead of a multi threaded design. So they were forced to add some hacks to load balance across multiple CPUs in the same physical load balancer in order to get the system to scale more (and there has been limitations over the years as to what could actually be distributed over multiple cores and what features were locked to a single core -- as time has gone on they have addressed most of those that I am aware of). One in particular I remember (which may be fixed now I'm not sure - would be curious to know if it was fixed how they fixed it) - was that each CPU core had it's own local memory with no knowledge of other CPUs - which means when doing HTTP caching  - each CPU had to cache the content individually - massively duplicating the cache and slashing the effectiveness of the memory you had on the box. This was further compounded by the 32-bitness of TMM itself in it's limited ability to address larger amounts of memory.

In any case the F5 design is somewhat arcane, they chose to bolt on software features early on instead of re-building the core. The strategy seems to have paid off though from a market share and profits standpoint, just from a technical standpoint it's kinda lame

To be fair there are some features in the multi threaded Citrix Netscaler that are not available that are available in the older legacy code.

Things I don't like about the Netscaler include their Java GUI which is slow as crap (they are working on a HTML 5 GUI - maybe that is in v10?), I mean it can literally take about 4 minutes to load up all of my server groups (Citrix term for F5 Pools). F5 I can load them in about half a second. I think the separation of services with regards to content switching on Citrix is, well pretty weird to say the least. If I want to do content filtering I have to have an internal virtual server and an external virtual server, the external one does the content filtering and forwards to the internal one. With F5 it was all in one (same for Zeus too). The terminology has been difficult to adjust to vs my F5 (and some Zeus) background.

I do miss the Priority Activation feature F5 has, there is no similar feature on Citrix as far as I know (well I think you can technically do it but the architecture of the Netscaler makes it a lot more complex). This feature allows you to have multiple pools of servers within a single pool at different priorities. By default the load balancer sends to the highest (or lowest? I forgot, it's been almost 2 years) group of servers, if that group fails then it goes to the next, and the next. I think you can even specify the minimum number of nodes to have in a group before it fails over entirely to the next group.

Not being able to re-use objects with the default scripting language just seems brain dead to me, so I am using the legacy scripting language.

So I do still miss F5 for some things, Zeus for some other things, though Netscaler is pretty neat in it's own respects. F5 obviously has a strong presence where I spent the last decade of my life in and around Seattle, being that it was founded and has it's HQ in Seattle. Still have a buncha friends over there. Some pretty amazing stories I've heard come out of that place, they grew so fast, it's hard to believe they are still in one piece after all they've been through, what a mess!

If you want to futz around with a Netscaler you have the option of downloading their virtual appliance (VPX) for free - I believe it has a default throughput limit of 1Mbit. Upgrades to as high as 3Gbps. Though the VPX is limited to two CPU cores last I recall. F5 and A10 have virtual appliances as well.

Crescendo did not have a virtual appliance, which is one of the reasons I wasn't particularly interested in perusing their offering back when they were around. The inside story of the collapse of Crescendo is the stuff geek movies are made out of. I won't talk about it here but it was just amazing to hear what happened.

The load balancing market is pretty interesting to see the different regions and where various players are stronger vs weaker. Radware for example is apparently strong over on the east coast but much less presence in the west. Citrix did a terrible job marketing the Netscaler for many years (a point they acknowledged to me), then there are those folks out there that still use Cisco (?!) which just confuses me.  Then there are the smaller guys like A10, Zeus, Brocade - Foundry networks (acquired by Brocade, of course) really did themselves a disservice when they let their load balancing technology sit for a good five years between hardware refreshes, they haven't been able to recover from that from what I've seen/heard. They tried to pitch me their latest iron a couple of years ago after it came out - only to find out that it didn't support SSL at the time - I mean come on -- of course they later fixed that lack of a feature but it was too late for my projects.

And in case you didn't know - Extreme used to have a load balancer WAY BACK WHEN. I never used it. I forget what it's called off the top of my head. Extreme also partnered with F5 in the early days and integrated F5 code into their network chipsets so their switches could do load balancing too (the last switch that had this was released almost a decade ago - nothing since). Though the code in the chipsets was very minimal and not useful for anything serious.

I've been holding onto this post to get confirmation from HP support, now that I have it, here goes something'

A commenter Karl made a comment that made my brain think about this another way, and it turns out I was stupid in my original assessment as for the system storing the data for the snapshot. For some reason I was thinking the system was storing the zeros, but rather it was storing the data the zeros were replacing.

So I'm a dumbass for thinking that. homer moment *doh*

BUT the feature still appears broken in that what should happen is if there is in fact 200GB of data written to the snapshot that implies that zeros overwrote 200GB worth of non zero'd data - and that data should of been reclaimed from the source volume. In this case it was not, only a tiny fraction (1,536MB of logical storage or 12x128MB chunks of data). So at the very least the bulk of the data usage should of been moved from the source volume to the snapshot (snapshot space is allocated separate from the source volume so it's easy to see which is using what). CURRENTLY the volume is showing 989GB of reserved space on the array with 120GB of written snapshot data and 140GB of file system data or around 260GB of total data which should come out to around 325GB of physical data in RAID 5 3+1, not 989thGB. But that space reclaiming technology is another feature thin copy reclamation. Which reclaims space from deleted snapshots.

So, sorry for being a dumbass for the original premise of the post, for some reason my brain got confused by the results of the tests, and it wasn't until Karl's comment that it made me think about it from the other angle.

I am talking to some more technical / non support people on Monday about this.

And thanks Karl

I got some good information from senior technical folks at 3PAR and it turns out the bulk of my problems are related to bugs in how one part reports raw space utilization (resulting in wildly inaccurate info), and a bug with regards to a space reclamation feature that was specifically disabled by a software patch on my array in order to fix another bug with space reclamation. So the fix for that is to update to a newer version of code which has that particular problem fixed for good(I hope?). I think I'd never get that kind of information out of the technical support team.

So in the end not much of a big issue after all, just confused by some bugs and functionality that was disabled and me being stupid.

END UPDATE

A lot of folks over the years have tried to claim I am a paid shill for 3PAR, or Extreme or whatever. All I can say is I'm not compensated by them for my posts in any direct way (maybe I get better discounts on occasion or something I'm not sure how that stuff factors in but in any case those benefits go to my companies rather than me).

I do knock them when I think they need to be knocked though. Here is something that made me want to knock 3PAR, well more than knock, more like kick in the butt, HARD and say W T F.

I was a very early adopter of the T-class of storage system, getting it in house just a couple months after it was released. It was the first system from them which had the thin built in - the thin reclamation and persistence technology integrated into the ASIC - only I couldn't use it because the software didn't exist at the time.

3PAR "Stay Thin" strategy - wasn't worth $100,000 in licensing for the extra 10% of additional capacity savings for my first big array.

That was kind of sad but it could of been worse - the competition that we were evaluating was Hitachi who had just released their AMS2000-series of products, literally about a month after the T-class was released. Hitachi had no thin provisioning support what-so-ever on the AMS2000 when it was launched. That came about seven months later. If you required thin provisioning at the time you had to buy a USP or (more common for this scenario due to costs at the time) a USP-V, which supported TP, and put the AMS2000 behind it. Hitachi refused to even give us a ballpark price as to the cost of TP on the AMS2000 whenever it was going to be released. I didn't need an exact price, just tell me is it going to be $5,000, or $25,000 or maybe $100,000 or more ? Should be a fairly simple process, at least from a customer perspective especially given they already had such licensing in place on their bigger platform. In the end I took that bigger platform's licensing costs(since they refused to give that to me too) and extrapolated what the cost might look like on the AMS line. I got the info from Storage Mojo's price list and basically took their price and cut it in half to take into account discounts and stuff. We ended up obviously not going for HDS so I don't know what it would of really cost us in the end.

OK, steering the tangent closer to the topic again..bear with me.

Which got me wondering - given it is an ASIC - and not a FPGA they really have to be damn sure it works when they ship product otherwise it can be an expensive proposition to replace the ASICs if there is a design problem with the chip, after all the CPUs aren't really in the data path of the stuff flowing through the system so it would be difficult to work around ASIC faults in software(if it was possible at all).

So I waited, and waited for the new thin stuff to come out, thinking since I had thin provisioning licensed already I would just install the new software and get the new features.

Then it was released - more than a year after I got the T400 but it came with a little surprise - additional licensing costs associated with the software - something nobody ever told me of (sounds like it was a last minute decision). If I recall right, for the system I had at the time if we wanted to fully license thin persistence it was going to be an extra $100,000 in software. We decided against it at the time, really wasn't worth the price for what we'd reclaim. Later on 3PAR offered to give us the software for free if we bought another storage array for disaster recovery (which we were planning to) - but the disaster recovery project got canned so we never got it.

Another licensing feature of this new software was in order to get to the good stuff, the thin persistence you had to license another product  - Thin Conversion whether you wanted it or not (I did not - really you might only need Thin Conversion if your migrating from a non thin storage system).

Fast forward almost two years and I'm at another company with another 3PAR, there was a thin provisioning licensing snafu with our system so for the past few months(and for the next few) I'm operating on an evaluation license which basically has all the features unlocked - including the thin reclamation tech. I had noticed recently that some of my volumes are getting pretty big - per the request of the DBA we have I agreed to make these volumes quite large - 400GB each, what I normally do is create the physical volume at 1 or 2TB (in this case 2TB), then I create a logical volume that is more in line with what the application actually needs(which may be as low as say 40GB for the database), then grow it on line as the space requirements increase.

3PAR's early marketing at least tried to communicate that you can do away with volume management altogether.  While certainly technically possible, I don't recommend that you take that approach. Another nice thing about volume management is being able to name the volumes with decent names, which is very helpful when working with moving snapshots between systems, especially with MPIO and multiple paths and multiple snapshots on one system, with LVM it's simple as can be, without - I really don't want to think about it. Only downside is you can't easily mount a snapshot back to the originating system because the LVM UUID will conflict and changing that ID is not (or was not, been a couple years since I looked into it) too easy, blocking access to the volume. Not a big deal though the number of times I felt I wanted to do that was once.

This is a strategy I came up with going back almost six years to my original 3PAR box and has worked quite well over the years. Originally, resizing was an off line operation since the kernel that we had at the time (Red Hat Enterprise 4.x) did not support on line file system growth, it does (and has) for a while now, I think since maybe 4.4/4.5 and certainly ever since v5.

Once you have a grasp as to the growth pattern of your application it's not difficult to plan for. Getting the growth plan in the first place could be complex though given the dedicate on write technology, you had to (borrowing a term from Apple here) think different. It obviously wasn't enough to just watch how much disk space was being consumed on average, you had to take into account space being written vs being deleted and how effective the file system was at re-utilizing deleted blocks. In the case of MySQL - being as inefficient as it is, you had to also take into account space utilization required by things like ALTER TABLE statements, in which MySQL makes a copy of the entire table with your change then drops the original. Yeah, real thin friendly there.

Given this kind of strategy it is more difficult to gauge just exactly how much your saving with thin provisioning, I mean on my original 3PAR I was about 300-400% over subscribed(which at the time was considered extremely high - I can't count the number of hours I spent achieving that), I think I recall at that conference I was at David Scott saying the average customer was 300% oversubscribed. On my current system I am 1300% over subscribed. Mainly because I got a bunch of databases and I make them all 2TB volumes, I can say with a good amount of certainty that they will probably never get to remotely 2TB in size but it doesn't affect me otherwise so I give it what I can (all my boxes on this array are VMware ESX 4.1 which of course has a 2TB limit - the bulk of these volumes are raw device mapped to leverage SAN-based snapshots as well as, to a lesser extent individually manage and monitor space and i/o metrics).

At the time my experience was compounded by the fact that I was still very green when it came to storage (I'd like to think I am more blue now at least whatever that might mean). Never really having dabbled much in it prior, choosing instead to focus on networking and servers. All big topics, I couldn't take them all on at once

So my point is - even though 3PAR has had this technology for a while now - I really have never tried it. In the past couple months I have run the Microsoft sdelete tool on the 3 windows VMs I do have to support my vCenter stuff(everything else is Linux) - but honestly I don't think I bothered to look to see if any space was reclaimed or not.

Now back on topic

Anyways, I have this one volume that was consuming about 300GB of logical space on the array when it had maybe 140GB of space written to the file system (which is 400GB). Obviously a good candidate for space reclamation, right? I mean the marketing claims you can gain 10% more space, in this case I'm gaining a lot more than that!

So I decided - hey how bout I write a basic script that writes out a ton of zeros to the file system to reclaim this space (since I recently learned that the kernel code required to do fancier stuff like fstrim [updated that post with new information at the end since I originally wrote it] doesn't exist on my systems). So I put a basic looping script in to write 100MB files filled with zeros from /dev/zero.

I watched it as it filled up the file system over time (I spaced out the writing as to not flood my front end storage connections), watching it reclaim very little space - at the end of writing roughly 200GB of data it reclaimed maybe 1-2GB from the original volume. I was quite puzzled to say the least. But that's not the topic of this post now is it.

I was shocked, awed, flabbergasted by the fact that my operation actually CONSUMED an additional 200GB of space on the system (space filled with zeros). Why did it do this? Apparently because I created a snapshot of the volume earlier in the day and the changes were being kept track of thus consuming the space. Never mind the fact that the system is supposed to drop the zeros even if it doesn't reclaim space - it doesn't appear to do so when there is a snapshot(s) on the volume, so the effects were a double negative - didn't reclaim any space from the original, and actually consumed a ton more space (more than 2x the original volume size) due to the snapshot.

Support claims minimal space was reclaimed by the system because I wrote files in 100MB blocks instead of 128MB blocks. I find it hard to believe out of 200GB of files I wrote that there was not more 128MB contiguous blocks of space of zeros. But I will try the test again with 128MB files on that specific volume after I can contact the people that are using the snapshot to delete the snapshot and re-create it to reclaim that 200GB of space. Hell I might as well not even use the snapshot and create a full physical copy of the volume.

Honestly I'm sort of at a loss for words as to how stupid this is. I have loved 3PAR through thick and thin for a long time (and I've had some big thicks over the years that I haven't written about here anyways..), but this one I felt compelled to. A feature so heavily marketed, so heavily touted on the platform is rendered completely ineffective when a basic function like snapshots is in use. Of course the documentation has nothing on this, I was looking through all the docs I had on the technology when I was running this test on Thursday and basically what it said was enable zero detection on the volume (disabled by default) and watch it work.

I've heard a lot of similar types of things (feature heavily touted but doesn't work under load or doesn't work period) on things like NetApp, EMC etc. This is a rare one for 3PAR in my experience at least. My favorite off the top of my head was NetApp's testing of an EMC CX-3 performance with snapshots enabled. That was quite a shocker to me when I first saw it. Roughly a 65% performance drop over the same system without snapshots.

Maybe it is a limitation of the ASIC itself - going back to my speculation about design issues and not being able to work around them in software. Maybe this limitation is not present in the V-class which is the next generation ASIC. Or maybe it is, I don't know.

HP Support says this behavior is as designed. Well I'm sure more than one person out there would agree it is a stupid design if so. I can't help but think it is a design flaw, not an intentional one - or a design aspect they did not have time to address in order to get the T-series of arrays out in a timely manor(I read elsewhere that the ASIC took much longer than they thought to design, which I think started in 2006 - and was at least partially responsible for them not having PCI express support when the ASIC finally came out). I sent them an email asking if this design was fixed in the V-Class, will update if they respond. I know plenty of 3PAR folks (current and former) read this too so they may be able to comment (anonymously or not..).

As for why more space was not reclaimed in the volume, I ran another test on Friday on another volume without any snapshots which should of reclaimed a couple hundred gigs but according to the command line it reclaimed nothing, support points me to logs saying 24GB was reclaimed, but that is not reflected in the command line output showing the raw volume size on the system. Still working with them on that one. My other question to them is why 24GB ? I wrote zeros to the end of the file system, there was  0 bytes left. I have some more advanced logging things to do for my next test.

While I'm here I might as well point out some of the other 3PAR software or features I have not used, let's see

• Adaptive optimization (sub LUN tiering - licensed separately)
• Full LUN-based automated tiering (which I believe is included with Dynamic optimization) - all of my 3PAR arrays to-date have had only a single tier of storage from a spindle performance perspective though had different tiers from RAID level perspectives
• Remote Copy - for the situations I have been in I have not seen a lot of value in array-based replication. Instead I use application based. The one exception is if I had a lot of little files to replicate, using block based replication is much more efficient and scalable. Array-based replication really needs application level integration, and I'd rather have real time replication from the likes of Oracle(not that I've used it in years, though I do miss it, really not a fan of MySQL) or MySQL then having to co-ordinate snapshots with the application to maintain consistency (and in the case of MySQL there really is no graceful way to take snapshots, again, unlike Oracle - I've been struggling recently with a race condition somewhere in an App or in MySQL itself which pretty much guarantees MySQL slaves will abort with error code 1032 after a simple restart of MySQL - this error has been shown to occur upwards of 15 minutes AFTER the slave has gotten back in sync with the master - really frustrating when trying to deal with snapshots and getting those kinds of issues from MySQL). I have built my systems, for the most part so they can be easily rebuilt so I really don't have to protect all of my VMs by replicating their data, I just have to protect/replicate the data I need in order to reconstruct the VM(s) in the event I need to.
• Recovery manager for Oracle (I licensed it once on my first system but never ended up using it due to limitations in it not being able to work with raw device maps on vmware - I'm not sure if they have fixed that by now)
• Recovery manager for all other products (SQL server, Exchange, and VMware)
• Virtual domains (useful for service providers I think mainly)
• Virtual lock (used to lock a volume from having data deleted or the volume deleted for a defined period of time if I recall right)
• Peer motion

3PAR Software/features I have used (to varying degrees)

• Thin Provisioning (for the most part pretty awesome but obviously not unique in the industry anymore)
• Dynamic Optimization (oh how I love thee) - the functionality this provides I think for the most part is still fairly unique, pretty much all of it being made possible by the sub disk chunklet-based RAID design of the system. Being able to move data around in the array between RAID levels, between tiers, between regions of the physical spindles themselves (inner vs outer tracks), really without any limit as to how you move it (e.g. no limitations like aggregates in the NetApp world), all without noticeable performance impact is quite amazing (as I wrote a while back I ran this process on my T400 once for four SOLID MONTHS 24x7 and nobody noticed).
• System Tuner (also damn cool - though never licensed it only used it in eval licenses) - this looks for hot chunklets and moves them around automatically. Most customers don't need this since the system balances itself so well out of the box. If I recall right, this product was created in response to a (big) customer's request mainly to show that it could be done, I am told very few license it since it's not needed. In the situations where I used it it ended up not having any positive(or negative) effect on the situation I was trying to resolve at the time.
• Virtual Copy (snapshots - both snapshots and full volume copies) - written tons of scripts to use this stuff mainly with MySQL and Oracle.
• MPIO Software for MS windows - worked fine - really not much to it, just a driver. Though there was some licensing fee 3PAR had to pay for MS for the software or development efforts they leveraged to build it - otherwise the drivers could of been free.
• Host Explorer (pretty neat utility that sends data back through the SCSI connection from the server to the array including info like OS version, MPIO version, driver versions etc - doesn't work on vSphere hosts because VMware hasn't implemented support for those SCSI commands or something)
• System Reporter - Collects a lot of data, though from a presentation perspective I much prefer my own cacti graphs
• vCenter Plugin for the array - really minimal set of functionality compared to the competition - a real weak point for the platform. Unfortunately it hasn't changed much in the almost two years since it was released - hoping it gets more attention in the future, or even in the present. As-is, I consider it basically useless and don't use it. I haven't taken advantage of the feature on my own system since I installed the software to verify that it's functional.
• Persistent Cache - an awesome feature in 4+ node systems that allows re-mirroring of cache to another node in the system in the event of planned or unplanned downtime on one or more nodes in the cluster (while I had this feature enabled - it was free with the upgrade to 2.3.1 on systems with 4 or more nodes I never actually had a situation where I was able to take advantage of it before I left the company with that system).
• Autonomic Groups - group volumes and systems together and make managing mappings of volumes to clusters of servers very easy. The GUI form of this is terrible and they are working to fix it. I literally practically wiped out my storage system when I first tried this feature using the GUI. It was scary the damage I did in the short time I had this(even more so given the number of years I've used the platform for). Fortunately the array that I was using was brand new and had really no data on it (literally). Since then - CLI for me, safer and much more clear as to what is going on. My friends over at 3PAR got a lot of folks involved over there to drive a priority plan to fix this functionality which they admit is lacking. What I did wipe out were my ESX boot volumes, so I had to re-create the volumes and re-install ESX. Another time I wiped out all of my fibre channel host mappings and had to re-establish those too. Obviously on a production system this would of resulted in massive data loss and massive downtime. Fortunately, again it was still at least 2 months from being a production system and had a trivial amount of data. When autonomic groups first came out I was on my T400 with a ton of existing volumes, migrating to use existing volumes to groups likely would of been disruptive so I only used groups for new resources, so I didn't get much exposure to the feature at the time.

That turned out to be A LOT longer than I expected.

This is probably the most negative thing I've said about 3PAR here. This information should be known though. I don't know how other platforms behave - maybe it's the same. But I can say in the nearly three years I have been aware of this technology this particular limitation has never come up in conversations with friends and contacts at 3PAR. Either they don't know about it either or it's just one of those things they don't want to admit to.

It may turn out that using SCSI UNMAP to reclaim space, rather than writing zeros is much more effective thus rendering the additional costs of thin licensing worth while. But not many things support that yet. As mentioned earlier, VMware specifically recommends disabling support for UNMAP in ESX 5.0 and has disabled it in subsequent releases because of performance issues.

Another thing that I found interesting, is that on the CLI itself, 3PAR specifically reccomends keeping Zero detection disabled unless your doing data migration because under heavy load it can cause issues -

Note: there can be some performance implication under extreme busy systems so it is recommended for this policy to be turned on only  during Fat to Thin and re-thinning process and be turned off during normal operation.

Which to some degree defeats the purpose? Some 3PAR folks have told me that information is out of date and only related to legacy systems. Which didn't really make sense since there are no legacy systems that support zero detection as it is hard wired into the ASIC. 3PAR goes around telling folks that zero detection on other platforms is no good because of the load it introduces but then says that their system behaves in a similar way. Now to give them credit I suspect it is still quite likely a 3PAR box can absorb that hit much better than any other storage platform out there, but it's not as if your dealing with a line rate operation, there clearly seems to be a limit as to what the ASIC can process. I would like to know what an extremely busy system looks like - how much I/O as a percentage of controller and/or disk capacity?

Bottom line - at this point I'm even more glad I didn't license the more advanced thinning technologies on my bigger T400 way back when.

I suppose I need to go back to reclaiming space the old fashioned way - data migration.

4,000+ words woohoo!

In any case they launched a new server line up today, which otherwise is not really news since who uses Sun/Oracle x86-64 boxes anyways? But I thought the news was interesting since it seems to include 4 x 10GbaseT ports on board as standard.

Rear of Sun Fire X4170 M3 Server

The Sun Fire X4170 M3 and the X4270 M3 systems both appear to have quad port 10GbaseT on the motherboard. I haven't heard of any other severs yet that have this as standard. Out of curioisity if you know of others I'd be interested to hear who they are.

The data sheet is kind of confusing, saying it has 4 onboard 10GbE ports but then it says Four 100/1,000/10 Base-T Ethernet ports in the network section below. Of course it was frequent to have 10/100/1000 BaseT before, so after seeing the physical rear of the system it seems convincing that they are using 10GbaseT.

Nice goin' Oracle.

It just seems wrong. I mean the networking stuff. Given NetApp's strong push for IP-based storage, one would think an entry level solution would simply have 2x48 port 10gig stackable switches, or whatever Cisco's equivalent is(maybe this is it).

This solution is supposed to provide scalability for up to 1,000 users - what those 1,000 users are actually doing I have no idea, does it mean VDI? Database? web site users? File serving users? ?????????????

It's also unclear in the article if this itself scales to that level or it just provides the minimum building blocks to scale to 1,000 users (I assume the latter) - and if so what does 1,000 user configuration look like? (or put another way how many users does the below picture support)

I'll be the first to admit I'm ignorant as to the details and the reason why Cisco needs 3 different devices with these things but whatever the reason seems major overkill for an entry level solution assuming the usage of IP-based storage.

The new entry level flex pod

The choice of a NetApp FAS2000 array seems curious to me - at least given the fact that it does not appear to support that Flex Cache stuff which they like to tout so much.

Anyways it's two people who sound experienced(I don't see information on their particular backgrounds) each talking up their respective solution. Two things really stood out to me:

• The guy pimping Microsoft was all about talking about a solution that doesn't exist yet ("it'll be fixed in the next version, just wait!")
• The guy pimping VMware was all about talking about how cost doesn't matter because VMware is the best.

I think they are both right - and both wrong.

It's good enough

I really believe that in Hyper-V's case and also in KVM/RHEV's case that for the next generation of projects these products will be "good enough" (in Hyper-V's case - whenever Windows 8 comes out) for a large(majority) number of use cases out there. I don't see Linux-centric shops considering Hyper-V or Windows-centric considering KVM/RHEV/etc so VMware will still obviously have a horse in the  race (as the pro-VMware person harps on in the debate).

Cost is becoming a very important issue

One thing that really got me the wrong way was when the pro-VMware person said this

Some people complain about VMware's pricing but those are not the decision makers, they are the techies. People who have the financial responsibility for SLAs and customers aren't going to bank on an unproven technology.

I'm sorry but that is just absurd. If cost wasn't an issue then the techies wouldn't be complaining about it because they know, first hand that it is an issue in their organizations. They know, first hand that they have to justify the purchase to those decision makers. The company I'm at now was in that same situation - the internal IT group could only get the most basic level of vSphere approved for purchase at the time for thier internal IT assets(this was a year or two or three ago). I hear them constantly complaining about the lack of things like vMotion, or shared storage etc. Cost was a major issue so the environment was built with disparate systems and storage and the cheap version of vSphere.

Give me an unlimited budget and I promise, PROMISE you will NEVER hear me complain about cost. I think the same is true of most people.

I've been there, more than once! I've done that exact same thing (Well in my case I managed to have good storage in most of the cases).

Those decision makers weigh the costs of maintaining that SLA with whatever solution they're going to provide. Breaking SLAs can be more cost effective then achieving them. Especially if they are absurdly high SLAs. I remember at one company I was at they signed all these high level SLAs with their new customers -- so I turned around and said - hey, in order to achieve those SLAs we need to do this laundry list of things. I think maybe 5-10% of the list got done until the budget ran out. You can continue to meet those high SLAs if your lucky, and don't actually have the ability to sustain failure and maintain uptime. More often than not such smaller companies prefer to rely on luck then doing things right.

Another company I was at had what could of been called a disaster in itself, during the same time I was working on a so-called disaster recovery project (no coincidence). Despite the disaster, at the end of the day the management canned the disaster recovery project (which everyone agreed if it was in place it would of saved a lot of everything had it been in place at the time of the disaster). It's not that budget wasn't approved - it was approved. The problem was management wanting to do another project that they massively under budgeted for and decided to cannibalize the budget from DR to give to this other pet project.

Yet another company I was at signed a disaster recovery contract with Sun Guard just to tick the check box to say they have DR. The catch was - the entire company knew up front before they signed - that they would never be able to utilize the service. IT WOULD NEVER WORK. But they signed it anyways because they needed a plan, and they didn't want to pay for a plan that would of worked.

VMware touting VM density as king

I've always found it interesting how VMware touts VM density, they show an automatic density advantage to VMware which automatically reduces VMware's costs regardless of the competition. This example was posted to one of their blogs a few days ago.

They tout their memory sharing, their memory ballooning, their memory compression all as things that can increase density vs the competiton.

My own experience with memory sharing on VMware at least with Linux is pretty simple - it doesn't work. It doesn't give results. Looking at one of my ESX 4.1 servers (yes, no ESXi here) which has 18 VMs on it and 101GB of memory in use, how much memory am I saving with the transparent page sharing?

3,161 MB - or about 3%. Nothing to write home about.

For production loads, I don't want to be in a situation where memory ballooning kicks in, or when memory compression kicks in, I want to keep performance high - that means no swapping of any kind from any system. Last thing I want is my VMs to start thrashing my storage with active swapping. Don't even think about swapping if your running Java apps either, once that garbage collection kicks in your VM will grind to a halt while it performs that operation.

I would like a method to keep the Linux buffer cache under control however, whether it is ballooning that specifically targets file system cache, or some other method, that would be a welcome addition to my systems.

Another welcome addition would be the ability to flag VMs and/or resource pools to pro-actively utilize memory compression (regardless of memory usage on the host itself). Low priority VMs, VMs that sit at 1% cpu usage most of the time, VM's where the added latency of compression on otherwise idle CPU cores isn't that important (again - stay away from actively swapping!). As a bonus provide the ability to limit the CPU capacity consumed by compression activities, such as limiting it to the resource pool that the VM is in, and/or having a per-host setting where you could say - set aside up to 1 CPU core or whatever for compression, if you need more than that, don't compress unless it's an emergency.

YAWA with regards to compression would be to provide me with compression ratios - how effective is the compression when it's in use? Recommend to me VMs that have low utilization that I could pro-actively reclaim memory by compressing these, or maybe only portions of the memory are worth compressing? The Hypervisor with the assistance of the vmware tools has the ability to see what is really going on in the guest by nature of having an agent there. The actual capability doesn't appear to exist now but I can't imagine it being too difficult to implement. Sort of along the lines of pro-actively inflating the memory balloon.

So, for what it's worth for me, you can take any VM density advantages for VMware off the table when it comes from a memory perspective. For me and VM density it's more about the efficiency of the code and how well it handles all of those virtual processors running at the same time.

Taking the Oracle VM blog post above, VMware points out Oracle supports only 128 VMs per host vs VMware at 512, good example - but really need to show how well all those VMs can work on the same host, how much overhead is there. If my average VM CPU utilization is 2-4% does that mean I can squeeze 512 VMs on a 32-core system (memory permitting of course)  -- when in theory I should be able to get around 640 - memory permitting again.

Oh the number of times I was logged into an Amazon virtual machine that was suffering from CPU problems only to see that 20-30% of the CPU usage was being stolen from the VM by the hypervisor. From the sar man page

%steal

Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

Not sure if Windows has something similar.

Back to costs vs Maturity

I was firmly in the VMware camp for many years, I remember purchasing ESX 3.1 (Standard edition - no budget for higher versions) for something like $3,500 for a two-socket license. I remember how cheap it felt at the time given the power it gave us to consolidate workloads. I would of been more than happy(myself at least) to pay double for what we got at the time. I remember the arguments I got in over VMware vs Xen with my new boss at the time, and the stories of the failed attempts to migrate to Xen after I left the company.

The pro-VMware guy in the original ZDNet debate doesn't see the damage VMware is doing to itself when it comes to licensing. VMware can do no wrong in his eyes. I'm sure there are plenty of other die hards out there that are in the same boat. The old motto of you never got fired for buying IBM right. I can certainly respect the angle though as much as it pains my own history to admit that I think the tides have changed and VMware will have a harder and harder time pitching it's wares in the future, especially if it keeps playing games with licensing on a technology which it's own founders (I think -- I wish I could find the article) predicted would become commodity by about now. With the perceived slow uptake of vSphere 5 amongst users I think the trend is already starting to form. The problem with the uptake isn't just the licensing of course, it's that for many situations there isn't a compelling reason to upgrade - it's good enough has set in.

I can certainly, positively understand VMware providing premium pricing for premium services, an Enterprise Plus Plus ..or whatever. But don't vary the price based on provisioned utilization that's just plain shooting yourself (and your loyal customers) in the feet. The provisioned part is another stickler for me - the hypervisor has the ability to measure actual usage, yet they stick their model to provisioned capacity - whether or not the VM is actually using the resource. It is a simpler model but it makes planning more complicated.

The biggest scam in this whole cloud computing era so many people think we're getting into is the vast chasm between provisioned vs utilized capacity. With companies wanting to charge you for provisioned capacity and customers wanting to over provision so they don't have to make constant changes to manage resources, knowing that they won't be using all that capacity up front.

The technology exists, it's just that few people are taking advantage of it and fewer yet have figured out how to leverage it (at least in the service provider space from what I have seen).

Take Terremark (now Verizon), a VMware-based public cloud provider (and one of only two partners listed on VMware's site for this now old program). They built their systems on VMware, they build their storage on 3PAR. Yet for this vCloud express offering there is no ability to leverage resource pools, no ability to utilize thin provisioning (from a customer standpoint). I have to pay attention to exactly how much space I provision up front, and I don't have the option to manage it like I would on my own 3PAR array.

Now Terremark has an enterprise offering that is more flexible and does offer resource pools, but this isn't available on their on demand offering. I still have the original quote Terremark sent me for the disaster recovery project I was working on at the time, it makes me want to either laugh or cry to this day. I have to give Terremark credit though at least they have an offering that can utilize resource pools, most others (well I haven't heard of even one - though I haven't looked recently) does not. (Side note: I hosted my own personal stuff on their vCloud express platform for a year so I know it first hand - it was a pretty good experience what drove me away primarily was their billing for each and every TCP and UDP port I had open on an hourly rate. Also good to not be on their platform anymore so I don't risk them killing my system if they see something I say and take it badly).

Obviously the trend in system design over recent years has bitten into the number of licenses that VMware is able to sell - and if their claims are remotely true - that 50% of the world's workloads are virtualized and of that they have 80% market share - it's going to be harder and harder to maintain a decent growth rate. It's quite a pickle they are in, customers in large part apparently haven't bought into the more premium products VMware has provided (that are not part of the Hypervisor), so they felt the pressure to increase the costs of the Hypervisor itself  to drive that growth in revenue.

Bottom Line

VMware is in trouble.

Simple as that.