TechOpsGuys.com Diggin' technology every day

February 27, 2010

Cisco UCS Networking falls short

Filed under: Networking,Virtualization — Tags: , , , , — Nate @ 4:33 am

UPDATED Yesterday when I woke up I had an email from Tolly in my inbox, describing a new report comparing the networking performance of the Cisco UCS vs the HP c Class blade systems. Both readers of the blog know I haven’t been a fan of Cisco for a long time(about 10 years, since I first started learning about the alternatives), and I’m a big fan of HP c Class (again never used it, but planning on it). So as you could imagine I couldn’t resist what it said considering the amount of hype that Cisco has managed to generate for their new systems(the sheer number of blog posts about it make me feel sick at times).

I learned a couple things from the report that I did not know about UCS before (I often times just write their solutions off since they have a track record of under performance, over price and needless complexity).

The first was that the switching fabric is external to the enclosure, so if two blades want to talk to each other that traffic must leave the chassis in order to do so, an interesting concept which can have significant performance and cost implications.

The second is that the current UCS design is 50% oversubscribed, which is what this report targets as a significant weakness of the UCS vs the HP c Class.

The mid plane design of the c7000 chassis is something that HP is pretty proud of(for good reason), capable of 160Gbps full duplex to every slot, totaling more than 5 Terrabits of fabric, they couldn’t help but take shots at IBM’s blade system and comment on how it is oversubscribed and how you have to be careful in how you configure the system based on that oversubscription when I talked to them last year.

This c7000 fabric is far faster than most high end chassis Ethernet switches, and should allow fairly transparent migration to 40Gbps ethernet when the standard arrives for those that need it. In fact HP already has 40Gbps Infiniband modules available for c Class.

The test involved six blades from each solution, when testing throughput of four blades both solutions performed similarly(UCS was 0.76Gbit faster). Add two more blades and start jacking up the bandwidth requirements. HP c Class scales linerally as the traffic goes up, UCS seems to scale lineraly in the opposite direction. End result is with 60Gbit of traffic being requested(6 blades @ 10Gbps), HP c Class managed to choke out 53.65Gbps, and Cisco UCS managed to cough up a mere 27.37Gbps. On UCS, pushing six blades at max performance actually resulted in less performance than four blades at max performance, significantly less. Illustrating serious weaknesses in the QoS on the system(again big surprise!).

The report mentions putting Cisco UCS in a special QoS mode for the test because without this mode performance was even worse. There is only 80Gbps of fabric available for use on the UCS(4x10Gbps full duplex). You can get a second fabric module for UCS but it cannot be used for active traffic, only as a backup.

UPDATE – A kind fellow over at Cisco took notice of our little blog here(thanks!!) and wanted to correct what they say is a bad test on the part of Tolly, apparently Tolly didn’t realize that the fabrics could be used in active-active(maybe that complexity thing rearing it’s head I don’t know). But in the end I believe the test results are still valid, just at an incorrect scale. Each blade requires 20Gbps of full duplex fabric in order to be non blocking throughout. The Cisco UCS chassis provides for 80Gbps of full duplex fabric, allowing 4 blades to be non blocking. HP by contrast allows up to three dual port Flex10 adapters per half height server which requires 120Gbps of full duplex fabric to support at line rate. Given each slot supports 160Gbps of fabric, you could get another adapter in there but I suspect there isn’t enough real estate on the blade to connect the adapter! I’m sure 120Gbps of ethernet on a single half height blade is way overkill, but if it doesn’t radically increase the cost of the system, as a techie myself I do like the fact that the capacity is there to grow into.

Things get a little more complicated when you start talking about non blocking internal fabric(between blades) and the rest of the network, since HP designs their switches to support 16 blades, and Cisco designs their fabric modules to support 8. You can see by the picture of the Flex10 switch that there are 8 uplink ports on it, not 16, but it’s pretty obvious that is due to space constraints because the switch is half width. END UPDATE

The point I am trying to make here isn’t so much the fact that HP’s architecture is superior to that of Cisco’s. It’s not that HP is faster than Cisco. It’s the fact that HP is not oversubscribed and Cisco is. In a world where we have had non blocking switch fabrics for nearly 15 years it is disgraceful that a vendor would have a solution where six servers cannot talk to each other without being blocked. I have operated 48-port gigabit swtiches which have 256 gigabits of switching fabric, that is more than enough for 48 systems to talk to each other in a non blocking way. There are 10Gbps switches that have 500-800 gigabits of switching fabric allowing 32-48 systems to talk to each other in a non blocking way. These aren’t exactly expensive solutions either. That’s not even considering the higher end backplane and midplane based system that run into the multiple terrabits of switching fabrics connecting hundreds of systems at line rates.

I would expect such a poor design to come from a second tier vendor, not a vendor that has a history of making networking gear for blade switches for several manufacturers for several years.

So say take it worst case, what if you want completely non blocking fabric from each and every system? For me I am looking to HP c Class and 10Gbs Virtual Connect mainly for inttra chassis communication within the vSphere environment. In this situation with a cheap configuration on HP, you are oversubscribed 2:1 when talking outside of the chassis. For most situations this is probably fine, but say that wasn’t good enough for you. Well you can fix it by installing two more 10Gbps switches on the chassis (each switch has 8x10GbE uplinks). That will give you 32x10Gbps uplink ports enough for 16 blades each having 2x10Gbps connections. All line rate, non blocking throughout the system. That is 320 Gigabits vs 80 Gigabits available on Cisco UCS.

HP doesn’t stop there, with 4x10Gbps switches you’ve only used up half of the available I/O slots on the c7000 enclosure, can we say 640 Gigabits of total non-blocking ethernet throughput vs 80 gigabits on UCS(single chassis for both) ? I mean for those fans of running vSphere over NFS, you could install vSphere on a USB stick or SD card and dedicate the rest of the I/O slots to networking if you really need that much throughput.

Of course this costs more than being oversubscribed, the point is the customer can make this decision based on their own requirements, rather than having the limitation be designed into the system.

Now think about this limitation in a larger scale environment. Think about the vBlock again from that new EMC/Cisco/VMware alliance. Set aside the fact that it’s horribly overpriced(I think mostly due to EMC’s side). But this system is designed to be used in large scale service providers. That means unpredictable loads from unrelated customers running on a shared environment. Toss in vMotion and DRS, you could be asking for trouble when it comes to this oversubscription stuff, vMotion (as far as I know) relies entirely on CPU and memory usage. At some point I think it will take storage I/O into account as well. I haven’t heard of it taking into account network congestion, though in theory it’s possible. But it’s much better to just have a non blocking fabric to begin with, you will increase your utilization, efficiency, and allow you to sleep better at night.

Makes me wonder how does Data Center Ethernet (whatever it’s called this week?) hold up under these congestion conditions that the UCS suffers from? Lots of “smart” people spent a lot of time making Ethernet lossless only to design the hardware so that it will incur significant loss in transit. In my experience systems don’t behave in a predictable manor when storage is highly constrained.

I find it kind of ironic that a blade solution from the world’s largest networking company would be so crippled when it came to the network of the system. Again, not a big surprise to me, but there are a lot of Cisco kids out there I see that drink their koolaid without thinking twice, and of course I couldn’t resist to rag again on Cisco.

I won’t bother to mention the recent 10Gbps Cisco Nexus test results that show how easily you can cripple it’s performance as well(while other manufacturers perform properly at non-blocking line rates), maybe will save that for another blog entry.

Just think, there is more throughput available to a single slot in a HP c7000 chassis than there is available to the entire chassis on a UCS. If you give Cisco the benefit of the second fabric module, setting aside the fact you can’t use it in active-active, the HP c7000 enclosure has 32 times the throughput capacity of the Cisco UCS. That kind of performance gap even makes Cisco’s switches look bad by comparison.

February 9, 2010

Why I hate the cloud

Filed under: Virtualization — Tags: — Nate @ 4:26 pm

Ugh, I hate all this talk about the cloud, for the most part what I can see is it’s a scam to sell mostly overpriced/high margin services to organizations who don’t know any better.  I’m sure there are plenty of organizations out there that have IT staff that aren’t as smart as my cat, but there are plenty that have people that are smarter too.

The whole cloud concept is sold pretty good I have to admit. It frustrates me so much I don’t know how properly express it. The marketing behind the cloud is such that it gives some people the impression that they can get nearly unlimited resources at their disposal, with good SLAs, good performance and pay pennies on the dollar.

It’s a fantasy. That reality doesn’t exist. Now sure the cost models of some incompetent organizations out there might be bad enough to the point that clouds make a lot of sense. But again there are quite a few that already have a cost effective way of operating. I suppose I am not the target customer, as every cloud provider I have talked to or seen cost analysis for has come in at a MINIMUM of 2.5-3x more expensive than doing it in house, going as high as 10x. Even the cheap crap that Amazon offers is a waste of money.

In my perspective, a public cloud(by which I mean an external cloud service provider, vs hosting “cloud” in house by way of virtual machines, grid computing and the like) has a few of use cases:

  1. Outsourced infrastructure for very small environments. I’m talking single digit servers here, low utilization etc.
  2. Outsourced “managed” cloud services, which would replace managed hosting(in the form of dedicated physical hardware) primarily to gain the abstraction layer from the hardware to handle things like fault tolerance and DR better. Again really only cost effective for small environments.
  3. Peak capacity processing – sounds good on paper, but you really need a scale-out application to be able to handle it, very few applications can handle such a situation gracefully. That is being able to nearly transparently shift compute resources to a remote cloud on demand for short periods of time to handle peak capacity. But I can’t emphasize enough the fact that the application really has to be built from the ground up to be able to handle such a situation. A lot of the newer “Web 2.0” type shops are building(or have built) such applications, but of course the VAST majority of applications most organizations will use were never designed with this concept in mind. There are frequently significant concerns surrounding privacy and security.

I’m sure you can extract other use cases, but in my opinion those other use cases assume a (nearly?) completely incompetent IT/Operations staff and/or management layers that prevent the organization from operating efficiently. I believe this is common in many larger organizations unfortunately, which is one reason I steer clear of them when looking for employment.

It just drives me nuts when I encounter someone who either claims the cloud is going to save them all the money in the world, or someone who is convinced that it will (but they haven’t yet found the provider that can do it).

Outside of the above use cases, I would bet money that for any reasonably efficient IT shop(usually involves a team of 10 or fewer people) can do this cloud thing far cheaper than any service provider would offer the service to them. And if a service provider did happen to offer at or below cost pricing I would call BS on them. Either they are overselling oversubscribed systems that they won’t be able to sustain, or they are buying customers so that they can build a customer base. Even what people often say is the low cost leader for cloud Amazon is FAR more expensive than doing it in house in every scenario I have seen.

Almost equally infuriating to me are those that believe all virtualization solutions are created equal, and that oh we can go use the free stuff(i.e. “free” Xen) rather than pay for vSphere. I am the first to admit that vSphere enterprise plus is not worth the $$ for virtually all customers out there, there is a TON of value available in the lower end versions of VMware. Much like Oracle, sadly it seems when many people think of VMware they immediately gravitate towards the ultra high end and say “oh no it’s too expensive!”. I’ve been running ESX for a few years now and have gotten by just fine without DRS, without host profiles, without distributed switches, without vMotion, without storage vMotion, the list goes on..! Not saying they aren’t nice features, but if you are cost conscious you often need to ask yourself while those are nice to have do you really need them. I’d wager frequently the answer is no.

February 4, 2010

Is Virtualisation ready for prime time?

Filed under: Virtualization — Tags: — Nate @ 12:06 pm

The Register asked that question and some people responded, anyone familiar ?

When was your first production virtualisation deployment and what did it entail? My brief story is below(copied from the comments of the first article, easier than re-writing it).

My first real production virtualization deployment was back in mid 2004 I believe, using VMware GSX I think v3.0 at the time(now called VMware server).

The deployment was an emergency decision that followed a failed software upgrade to a cluster of real production servers that was shared by many customers. The upgrade was supposed to add support for a new customer that was launching within the week(they had already started a TV advertising campaign). Every attempt was made to make the real deployment work but there were critical bugs and it had to get rolled back, after staying up all night working on it people started asking what we were going to do next.

One idea(forgot who maybe it was me) was to build a new server with vmware and transfer the QA VM images to it(1 tomcat web server, 1 BEA weblogic app server, 1 win2k SQL/IIS server, the main DB was on Oracle and we used another schema for that cluster on our existing DB) and use it for production, that would be the fastest turnaround to get something working. The expected load was supposed to be really low so we went forward. I spent what felt like 60 of the next 72 hours getting the systems ready and tested over the weekend with some QA help, and we launched on schedule on the following Monday.

Why VMs and not real servers? Well we already had the VM images, and we were really short on physical servers, at least good ones anyways. Back then building a new server from scratch was a fairly painful process, though not as painful as integrating a brand new environment. What would usually take weeks of testing we pulled off in a couple of days. I remember one of the tough/last issues to track down was a portion of the application failing due to a missing entry in /etc/hosts (a new portion of functionality that not many were aware of).

The second time I’ve managed to make The Register(yay!), the first would be a response to my Xiotech speculations a few months back.

November 24, 2009

81,000 RAID arrays

Filed under: Storage,Virtualization — Tags: , , — Nate @ 2:56 pm

I keep forgetting to post about this, I find this number interesting myself. It is the number of mini RAID arrays on my storage system, which has 200 spindles, which comes out to about 400 RAID arrays per disk! Why so many? Well it allows for maximum distribution of storage space and I/O across the system as well as massively parallel RAID rebuilds as every disk in the system will participate when a disk fails, which leads to faster rebuild times and much better service times during rebuild.

While 3PAR talks a lot about their mini RAID arrays(composed of virtual 256MB disks) it turns out there really isn’t an easy way to query how many there are, I suppose because they expect it to be so abstracted that you should not care. But I like to poke around if you haven’t noticed already!

The little script to determine this number is:

#!/bin/bash

export ARRAYS_TOTAL=0
export ARRAY="mrt"
echo "showld -d" | ssh $ARRAY | grep cage | while read line;
do
        export RAWSIZE=`echo $line | awk '{print $7}'`;
        export LD_NAME=`echo $line | awk '{print $2}'`;
        export SETSIZE=`echo $line | awk '{print $10}'`;
        export ARRAYS=`echo "${RAWSIZE}/256/${SETSIZE}" | bc`;
        export ALL_ARRAYS=`echo "${ARRAYS_TOTAL}+${ARRAYS}" | bc `;
        export ARRAYS_TOTAL="$ALL_ARRAYS"; echo "NAME:${LD_NAME} Raw Size:${RAWSIZE}  Set Size:${SETSIZE} Micro arrays in LD:${ARRAYS}  Total Micro arrays so far:${ALL_ARRAYS}";
done

Hopefully my math is right..

Output looks like:

NAME:log2.0 Raw Size:40960  Set Size:2 Micro arrays in LD:80  Total Micro arrays so far:80
NAME:log3.0 Raw Size:40960  Set Size:2 Micro arrays in LD:80  Total Micro arrays so far:160
NAME:pdsld0.0 Raw Size:49152  Set Size:2 Micro arrays in LD:96  Total Micro arrays so far:256
[..]
NAME:tp-7-sd-0.242 Raw Size:19968  Set Size:6 Micro arrays in LD:13  Total Micro arrays so far:81351
NAME:tp-7-sd-0.243 Raw Size:19968  Set Size:6 Micro arrays in LD:13  Total Micro arrays so far:81364

Like the mini RAID arrays the logical disks (the command above is showld or show logical disks) are created/maintained/deleted automatically by the system, another layer of abstraction that you really never need to concern yourself with.

The exact number is 81,364 which is up from about 79,000 in June of this year. To me at least it’s a pretty amazing number when I think about it, 80,000+ little arrays working in parallel, how does the system keep track of it all?

3PAR isn’t unique in this technology though I think maybe they were first. I believe Compellent has something similar, and Xiotech constructs RAID arrays at the disk head level, which I find very interesting, I didn’t know it was possible to “target” areas of the disk as specifically as the head. I think of these three implimentations though the 3PAR one is the most advanced because it’s implimented in hardware(Compellent is software), and it’s much more granular(400 per disk in this example, Xiotech would have up to 8 per disk I think).

The disks are not full yet either, they are running at about ~87% of capacity, so maybe room for another 10,000 RAID arrays on them or something..

I learned pretty quick there’s a lot more to storage than just the number/type of disks..

(Filed under virtualization as well since this is a virtualized storage post)

November 17, 2009

HP VirtualConnect for Dummies

Filed under: Networking,Storage,Virtualization — Tags: , , , — Nate @ 5:27 pm

Don’t know what VirtualConnect is? Check this e-book out. Available to the first 2,500 people that register. I just browsed over it myself it seems pretty good.

I am looking forward to using the technology sometime next year(trying to wait for the 12-core Opterons before getting another blade system). Certainly looks really nice on paper, and the price is quite good as well compared to the competition. It was first introduced I believe in 2006 so it’s fairly mature technology.

November 6, 2009

Thin Provisioning strategy with VMware

Filed under: Storage,Virtualization — Tags: , , , — Nate @ 3:44 pm

Since the announcement of thin provisioning built into vSphere I have seen quite a few blog posts on how to take advantage of it but haven’t seen anything that matches my strategy which has served me well utilizing array-based thin provisioning technology. I think it’s pretty foolproof..

The man caveat is that I assume you have a decent amount of storage available on your system, that is your VMFS volumes aren’t the only thing residing on your storage. On my current storage array,written VMFS data accounts for maybe 2-3 % of my storage. On the storage array I had at my last company it was probably 10-15%. I don’t believe in dedicated storage arrays myself. I prefer nice shared storage systems that can sustain random and sequential I/O from any number of hosts and distributed that I/O across all of the resources for maximum efficiency.  So my current array has most of it’s space set aside for a NFS cluster, and then there is a couple dozen terabytes set aside for SQL servers and VMware. The main key is being able to share the same spindles across dozens or even hundreds of LUNs.

There has been a lot of debate over the recent years about how best to size your VMFS volumes. The most recent data I have seen suggests somewhere between 250GB and 500GB. There seems to be unanimous opinion out there not to do something crazy and use 2TB volumes. The exact size depends on your setup. How many VMs, how many hosts, how often you use snapshots, how often you do vMotion, as well as the amount of I/O that goes on. The less of all of those the larger the volume can potentially be.

My method is so simple. I chose 1TB as my volume sizes, thin provisioned of course.  I utilize the default lazy zero VMFS mode and do not explicitly turn on thin provisioning on any VMDK files. There’s no real point if you already have it in the array. So I create 1TB volumes, and I begin creating VMs on them. I try to stop when I get to around 500GB of allocated(but not written) space. That is VMware thinks it is using 500GB, but it may only be using 30GB. This way I know, the system will never use more than 500GB. Pretty simple. Of course I have enough space in reserve that if something crazy were to happen the volume could grow to 500GB and not cause any problems. Even with my current storage array operating in the neighborhood of 89% of total capacity, that still leaves me with several terabytes of space I can use in an emergency.

If I so desire I can go beyond the 500GB at any time without an issue. If I chose not to then I haven’t wasted any space because nothing is written to those blocks. My thin provisioning system is licensed based on written data, so if I have 10TB of thin provisioning on my system I can, if I want create 100TB of thin provisioned volumes, provided I don’t write more than 10TB to them. So you see there really is no loss in making a larger volume when the data is thin provisioned on the array. Why not make it 2TB or even bigger? Well really I can’t see a time when I would EVER want a 2TB VMFS volume which is why I picked 1TB.

I took the time in my early days working with thin provisioning to learn the growth trends of various applications and how best to utilize them to get maximum gain out of thin provisioning.  With VMs that means having a small dedicated disk for OS and swap, and any data resides on other VMDKs or preferably on a NAS or for databases on raw devices(for snapshot purposes). Given that core OSs don’t grow much there isn’t much space needed(I default to 8GB) for the OS, and I give the OS a 1GB swap partition.  For additional VMDKs or raw devices I always use LVM. I use it to assist me in automatically detecting what devices a particular volume are on, I use it for naming purposes, and I use it to forcefully contain growth. Some applications are not thin provisioning friendly but I’d like to be able to expand the volume on demand without an outage. Online LVM resize and file system resize allows this without touching the array. It really doesn’t take much work.

On my systems I don’t really do vMotion(not licensed), I very rarely use VMFS snapshots(few times a year), the I/O on my VMFS volumes is tiny despite having 300+ VMs running on them. So in theory I probably could get away with 1TB or even 2TB VMFS volume sizes, but why lock myself into that if I don’t have to? So I don’t.

I also use dedicated swap VMFS volumes so I can monitor the amount of I/O going on with swap from an array perspective. Currently I have 21 VMware hosts connected to our array totalling 168 CPU cores, and 795GB of memory. Working to retire our main production VMware hosts, many of which are several years old(re-purposed from other applications). Now that I’ve proven how well it can work on existing hardware and the low cost version the company is ready to gear up a bit more and commit more resources to a more formalized deployment utilizing the latest hardware and software technology. You won’t catch me using the enterprise plus or even the enterprise version of VMware though, cost/ benefit isn’t there.

November 3, 2009

The new Cisco/EMC/Vmware alliance – the vBlock

Filed under: Storage,Virtualization — Tags: , , , , , , , — Nate @ 6:04 pm

Details were released a short time ago thanks to The Register on the vBlock systems coming from the new alliance of Cisco and EMC, who dragged along Vmware(kicking and screaming I’m sure). The basic gist of it is to be able to order a vBlock and have it be a completely integrated set of infrastructure ready to go, servers and networking from Cisco, storage from EMC, and Hypervisor from VMware.

vBlock0 consists of rack mount servers from Cisco, and unknown EMC storage, price not determined yet

vBlock1 consists 16-32 blade servers from Cisco and EMC CX4-480 storage system. Price ranges from $1M – 2.8M

vBlock2 consists of 32-64 blade servers from Cisco and an EMC V-MAX. Starting price $6M.

Sort of like FCoE, sounds nice in concept but the details fall flat on their face.

First off is the lack of choice. That is Cisco’s blades are based entirely on the Xeon 5500s, which are, you guessed it limited to two sockets. And at least at the moment limited to four cores. I haven’t seen word yet on compatibility with the upcoming 8-core cpus if they are socket/chip set compatible with existing systems or not(if so, wonderful for them..). Myself I prefer more raw cores, and AMD is the one that has them today(Istanbul with 6 cores, Q1 2010 with 12 cores). But maybe not everyone wants that so it’s nice to have choice. In my view HP blades win out here for having the broadest selection of offerings from both Intel and AMD. Combine that with their dense memory capacity(16 or 18 DIMM slots on a half height blade), allows you up to 1TB of memory in a blade chassis in an afforadable confiugration using 4GB DIMMs. Yes Cisco has their memory extender technology but again IMO at least with a dual socket Xeon 5500 that it is linked to the CPU core:memory density is way outta whack. It may make more sense when we have 16, 24, or even 32 cores on a system using this technology. I’m sure there are niche applications that can take advantage of it on a dual socket/quad core configuration, but the current Xeon 5500 is really holding them back with this technology.

Networking, it’s all FCoE based, I’ve already written a blog entry on that, you can read about my thoughts on FCoE here.

Storage, you can see how even with the V-MAX EMC hasn’t been able to come up with a storage system that can start on the smaller end of the scale, something that is not insanely unaffordable to 90%+ of the organizations out there. So on the more affordable end they offer you a CX4. If you are an organization that is growing you may find yourself outliving this array pretty quickly. You can add another vBlock, or you can rip and replace it with a V-MAX which will scale much better, but of course the entry level pricing for such a system makes it unsuitable for almost everyone to try to start out with even on the low end.

I am biased towards 3PAR of course as both of the readers of the blog know, so do yourself a favor and check out their F and T series systems, if you really think you want to scale high go for a 2-node T800, the price isn’t that huge, the only difference between a T400 and a T800 is the backplane. They use “blocks” to some extent, blocks being controllers(in pairs, up to four pairs), disk chassis(40 disks per chassis, up to 8 per controller pair I think). Certainly you can’t go on forever, or can you? If you don’t imagine you will scale to really massive levels go for a T400 or even a F400.  In all cases you can start out with only two controllers the additional cost to give you the option of an online upgrade to four controllers is really trivial, and offers nice peace of mind. You can even go from a T400 to a T800 if you wanted, just need to switch out the back plane (downtime involved). The parts are the same! the OS is the same! How much does it cost? Not as much as you would expect. When 3PAR announced their first generation 8-node system 7 years ago, entry level price started at $100k. You also get nice things like their thin built in technology which will allow you to run those eager zeroed VMs for fault tolerance and not consume any disk space or I/O for the zeros. You can also get multi level synchronous/asynchronous replication for a fraction of the cost of others. I could go on all day but you get the idea. There are so many fiber ports on the 3PAR arrays that you don’t need a big SAN infrastructure just hook your blade enclosures directly to the array.

And as for networking hook your 10GbE Virtual Connect switches on your c Class enclosures to your existing infrastructure. I am hoping/expecting HP to support 10GbaseT soon, and drop the CX4 passive copper cabling. The Extreme Networks Summit X650 stands alone as the best 1U 10GbE (10GbaseT or SFP+) switch on the market. Whether it is line rate, or full layer 3, or high speed stacking, or lower power consuming 10GbaseT vs fiber optics,  or advanced layer 3 networking protocols to simplify management,  price and ease of use — nobody else comes close. If you want bigger check out the Black Diamond 8900 series.

Second you can see with their designs that after the first block or two the whole idea of a vBlock sort of falls apart. That is pretty quickly your likely to just be adding more blades(especially if you have a V-MAX), rather than adding more storage and more blades.

Third you get the sense that these aren’t really blocks at all. The first tier is composed of rack mount systems, the second tier is blade systems with CX4, the third tier is blade systems with V-MAX. Each tier has something unique which hardly makes it a solution you can build as a “block” as you might expect from something called a vBlock. Given the prices here I am honestly shocked that the first tier is using rack mount systems. Blade chassis do not cost much, I would of expected them to simply use a blade chassis with just one or two blades in it. Really shows that they didn’t spend much time thinking about this.

I suppose if you treated these as blocks in their strictest sense and said yes we won’t add more than 64 blades to a V-MAX, and add it like that you could get true blocks, but I can imagine the amount of waste doing something like that is astronomical.

I didn’t touch on Vmware at all, I think their solution is solid, and they have quite a bit of choices. I’m certain with this vBlock they will pimp the enterprise plus version of software, but I really don’t see a big advantage of that version with such a small number of physical systems(a good chunk of the reason to go to that is improved management with things like host profiles and distributed switches). As another blogger recently noted, Vmware has everything to lose out of this alliance, I’m sure they have been fighting hard to maintain their independence and openness, this reeks of the opposite, they will have to stay on their toes for a while when dealing with their other partners like HP, IBM, NetApp, and others..

September 14, 2009

Fix hanging vmware tools on linux

Filed under: Virtualization — Tags: , — Nate @ 5:48 pm

I can’t be the only one who has come across this, back in early June I filed a support case with VMware around the fact that roughly 90% of the time when the latest version of vmware-tools that shipped with vSphere loaded on my CentOS 5 systems it would hang part way through, if I logged into the console I and just pressed <enter> it would continue loading. Naturally the Tier 1 support rep was fairly useless, wanting me to do some stupid things to get more debug information.

I went off on my own and traced down the problem to the vmware-config-tools.pl script towards the end of the script at around line 11,600, where it tries to make a symlink. If I disable the offending code the problem stops(the link it’s trying to create is in fact already there):

sub symlink_icudt38l {
my $libdir = db_get_answer('LIBDIR');
install_symlink($libdir . '/icu', $gRegistryDir . '/icu');
}

If your interested in the strace output:

[..]
[pid  7228] <... read resumed> "", 4096) = 0
[pid  7228] --- SIGCHLD (Child exited) @ 0 (0) ---
[pid  7228] fstat(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid  7228] close(4)                    = 0
[pid  7228] rt_sigaction(SIGHUP, {SIG_IGN}, {SIG_IGN}, 8) = 0
[pid  7228] rt_sigaction(SIGINT, {SIG_IGN}, {SIG_DFL}, 8) =  0
[pid  7228] rt_sigaction(SIGQUIT, {SIG_IGN},  {SIG_DFL}, 8) = 0
[pid  7228] wait4(7244, [{WIFEXITED(s)  && WEXITSTATUS(s) == 0}], 0, NULL) = 7244
[pid   7228] rt_sigaction(SIGHUP, {SIG_IGN}, NULL, 8) = 0
[pid   7228] rt_sigaction(SIGINT, {SIG_DFL}, NULL, 8) = 0
[pid   7228] rt_sigaction(SIGQUIT, {SIG_DFL}, NULL, 8) = 0
[pid  7228] lstat("/etc/vmware-tools/icu", {st_mode=S_IFLNK|0777,  st_size=25, ...}) = 0
[pid  7228] read(0,

The last line there the system is waiting for input, when I hit <enter> it continues loading.

The support case sat..and sat..and sat. Then a couple of weeks ago some manager called me up and wanted to know how the case was doing. I guess they didn’t spend any time on it at all. I told him I already found a workaround, and he said because of that they were going to work up a KB entry on it then close the case. Then another week passes and I get an email saying OH! We see you found a workaround, we’ll forward that to engineering and get back to you. Yes the workaround I sent on JUNE 16.

So hope this helps someone, I’ll update this when/if they get a KB entry out on it. It’s certainly saved me a lot of time, it is very annoying to have to connect to each and every system to press enter to get it to continue to boot to workaround this bug.

September 10, 2009

Where is the serial console in ESXi

Filed under: Monitoring,Virtualization — Tags: , — Nate @ 8:28 am

Back to something more technical I suppose. I was kind of surprised and quite disappointed when vSphere was released with an ESXi that did not have serial console support. I can understand not having it in the first iteration but I think it’s been over a year since ESXi was first released and still no serial console support? I guess it shows how Microsoft-centric VMware has been(not forgetting that Windows 2003 introduced an emergency console on the serial port, though I haven’t known anyone that has used it).

Why serial console? Because it’s faster and easier to access. Most good servers have the ability to access a serial console over SSH, be it from HP, or Dell, or Rackable, probably IBM too. Last I checked Sun only supported telnet, not ssh, though that may of changed recently. A long time ago with HP iLO v1 HP allowed you to access the “VGA” console via SSH, using the remcons command, this vanished in iLO v2(unless they added it back in recently I haven’t had an iLO 2 system in about 1.5 years). If your dealing with a system that is several networks away, it is so much faster to get to the console with SSH then bouncing around with the web browser and fooling with browser plug ins to get to the VGA console.

Also serial console has the ability(in theory anyways) to log what you get on the serial console to a syslog or other kind of server(most console/terminal servers can do this) since it is all text. I haven’t yet seen a DRAC or an ILO that can do this that would be a nice feature to have.

ESX(non i) does support serial console though enabling it isn’t too straight forward, but at least it can be done.

Come on VMware for your next release of ESXi please add full serial console support, to be able to not only access the console while it’s booted but be able to install over serial console as well. Thanks in advance, not holding my breath!

August 28, 2009

vSphere Storage vMotion for free

Filed under: Storage,Virtualization — Tags: , , — Nate @ 2:47 pm

OK this is slightly obvious but it may not be to everyone.  Among my favorite new abiltiies in vSphere is the evaluation mode. In ESX 3.x, this evaluation mode was fairly locked down, in order to do anything you had to have a paid license, or a VAR that could give you a temporary license. But not with vSphere.

My company went through a storage array migration earlier this year, I had initially started deploying VMs on top of NFS last year on the previous array, when we got the new array all new VMs went to it directly, the old VMs hung around. My plan was basically to re-install them from scratch onto the new array so I could take advantage of the thin provisoning, I could save upwards of 90% of my space with thin provisioning so I didn’t just want to copy the data files over to the VMFS volume(our Thin provisioning is dedicate on write). With our new array came a pair of NAS heads from another company, in order to evacuate the old array I did move those data files over to the NFS side of the storage system as a holding area until I could get the time to re-install them onto VMFS volumes.

Then vSphere came out and the clouds parted. The evaluation mode was fully unlocked, every feature(that I know of) was available for use free for 60 days. After a few fairly quick tests I started migrating my production hosts as quickly as I could to them even before I had my replacement license keys, since I had 60 days to get them.  I setup an evaluation copy of vCenter, and hooked everything up. My first real exposure to vCenter. And I took the opportunity to use the free storage vMotion to migrate those VMs from the NFS data store to the VMFS data store in a “thin” way.

I don’t anticipate having or needing to use Storage vMotion often, but it’s nice to know that if I do need it, I can just fire up a new ESX system under a evaluation license, do my Storage vMotions to my heart’s content and then shut the new box down again. Since all of my systems boot from SAN, I could even do the same in-place, evacuate one system, unmap the original LUN, create a new LUN, install ESX on it, do the basics to get it configured, do the vMotions that I need, and then reboot the host, remove the new LUN, re-instate the old LUN and off we go again. Quite a few more steps but certainly worth it for me, if I only think I will need it once or twice per year.

We bought vCenter standard edition not too long ago, and I still have a couple vSphere hosts running in evaluation mode even now, in case I want to play with any of the more advanced features, I’ve done a few vMotions and stuff to shift VMs back onto freshly installed vSphere hosts. I only have 1 production ESX 3.5 system left, and it will stay 3.5 for a while at least because of the snapshot issues in vSphere.

Personally my four favorite things in vSphere are: Round Robin MPIO, ESXi boot from SAN, the Essentials license pricing, and the expanded evaluation functionality.

I really didn’t have much interest in most of the other things that VMware has been touting, I have simple requirements. And perhaps as a side effect I’ve had fewer problems then most serious ESX users I’ve talked with, have heard lots of horror stories about snapshots and VCB, and Dave here has even had some issues with zombie VMs and vMotion, others having issues with SCSI reservations etc.  I haven’t had any of that I guess because I almost never use any of that functionality. The core stuff is pretty solid, still have yet to see a system crash even. Good design? Luck? both? Other?

« Newer Posts — Older Posts »

Powered by WordPress