Dec/115
Red Hat Bringing back UML ?
TechOps Guy: Nate
User mode linux was kind of popular many years ago especially with the cheap virtual hosting crowd, but interest seemed to die off a while ago, with what seems to be a semi-official page for user mode linux not being updated since the Fedora Core 5 days which was around 2006.
Red hat apparently just released RHEL 6.2, and among the features, is something that looks remarkably similar to UML -
Linux Containers
• Linux containers provide a flexible approach to application runtime containment on bare-metal without the need to fully virtualize the workload. This release provides application level containers to separate and control the application resource usage policies via cgroup and namespaces. This release introduces basic management of container life-cycle by allowing for creation, editing and deletion of containers via the libvirt API and the virt-manager GUI.
• Linux Containers provides a means to run applications in a container, a deployment model familiar to UNIX administrators. Also provides container life-cycle management for these containerized applications through a graphical user interface (GUI) and user space utility (libvirt).
• Linux Containers is in Technology Preview at this time.
Which seems to be basically an attempt at a clone of Solaris containers. Seems like a strange approach for Red Hat to take given the investment in KVM. I struggle to think of a good use case for Linux containers over KVM.
Red hat also has enhanced KVM quite a bit, this update sort of caught my eye
Virtual CPU timeslice sharing for multiprocessor guests is a new feature in Red Hat Enterprise Linux 6.2. Scheduler changes within the kernel now allow for virtual CPUs inside a guest to make more efficient use of the timeslice allocated to the guest, before processor time is yielded back to the host. This change is especially beneficial to large SMP systems that have traditionally experienced guest performance lag due to inherent lock holder preemption issues. In summary, this new feature eliminates resource consuming system overhead so that a guest can use more of the CPU resources assigned to them much more efficiently.
No informations on specifics as far as what constitutes a “large” system or how many virtual CPUs were provisioned for a given physical CPU etc. But it’s interesting to see, I mean it’s one of those technical details in hypervisors that you just can’t get an indication from by viewing a spec sheet or a manual or something. Such things are rarely talked about in presentations as well. I remember being at a VMware presentation a few years ago where they mentioned they could of enabled 8-way SMP on ESX 3.x, it was apparently an undocumented feature, but chose not to because the scheduler overhead didn’t make it worth while.
Red Hat also integrated the beta of their RHEV 3 platform, I’m hopeful this new platform develops into something that can better compete with vSphere. Though their website is really devoid of information at this point which is unfortunate.
They also make an erroneous claim that RHEV 3 crushes the competition by running more VMs than anyone else and site a SPECvirt benchmark as the proof. While the results are impressive they aren’t really up front with the fact that the hardware more than anything else drove the performance with 80 x 2.4Ghz CPU cores, 2TB of memory and more than 500 spindles. If you look at the results on a more level playing field the performance of RHEV 3 and vSphere is more in line. RHEV still wins, but not by a crushing amount. I really wish these VM benchmarks gave some indication as to how much disk I/O was going on. It is interesting to see all the tuning measures that are disclosed, gives some good information on settings to go investigate maybe they have broader applications than synthetic benchmarking.
Of course performance is only a part of what is needed in a hypervisor, hopefully RHEV 3 will be as functional as it is fast.
There is a Enterprise Hypervisor Comparison released recently by VMGuru.nl, which does a pretty good job at comparing the major hypervisors, though does not include KVM. I’d like to see more of these comparisons from other angles, if you know of more guides let me know.
One thing that stands out a lot is OS support, it’s strange to me how VMware can support so many operating systems but other hypervisors don’t. Is this simply a matter of choice? Or is the VM technology VMware has so much better that it allows them to support the broader number of guest operating systems with little/no effort on their part? Or both ? I mean Hyper-V not supporting Windows NT ? How hard can it be to support that old thing? Nobody other than VMware supporting Solaris ?
I’ve talked off and on about KVM, as I watch and wait for it to mature more. I haven’t used KVM yet myself. I will target RHEV 3, when it is released, to try and see where it stands.
I’m kind of excited. Kind of because breaking up with VMware after 12 years is not going to be easy for me 
Nov/1111
Oracle throws in Xen virtualization towel?
TechOps Guy: Nate
This just hit me a few seconds ago and it gave me something else to write about so here goes.
Oracle recently released Solaris 11, the first major rev to Solaris in many many years. I remember using Solaris 10 back in 2005, wow it’s been a while!
They’re calling it the first cloud OS. I can’t say I really agree with that, vSphere, and even ESX before that has been more cloudy than Solaris for many years now, and remains today.
While their Xen-based Oracle VM is still included in Solaris 11, the focus clearly seems to be Solaris Zones, which, as far as I know is a more advanced version of User mode linux (which seems to be abandoned now?).
Zones, and UML are nothing new, Zones having been first released more than six years ago. It’s certainly a different approach to a full hypervisor approach so has less overhead, but overall I believe is an outdated approach to utility computing (using the term cloud computing makes me feel sick).
Oracle Solaris Zones virtualization scales up to hundreds of zones per physical node at a 15x lower overhead than VMware and without artificial limits on memory, network, CPU and storage resources.
It’s an interesting strategy, and a fairly unique one in today’s world, so it should give Oracle some differentiation. I have been following the Xen bandwagon off and on for many years and never felt it a compelling platform, without a re-write. Red Hat, SuSE and several other open source folks have basically abandoned Xen at this point and now it seems Oracle is shifting focus away from Xen as well.
I don’t see many new organizations gravitating towards Solaris zones that aren’t Solaris users already (or at least have Solaris expertise in house), if they haven’t switched by now…
New, integrated network virtualization allows customers to create high-performance, low-cost data center topologies within a single OS instance for ultimate flexibility, bandwidth control and observability.
The terms ultimate flexibility and single OS instance seem to be in conflict here.
The efficiency of modern hypervisors is to the point now where the overhead doesn’t matter in probably 98% of cases. The other 2% can be handled by running jobs on physical hardware. I still don’t believe I would run a hypervisor on workloads that are truely hardware bound, ones that really exploit the performance of the underlying hardware. Those are few and far between outside of specialist niches these days though, I had one about a year and a half ago, but haven’t come across one since.
Nov/113
Mass defections away from Vmware coming?
TechOps Guy: Nate
I have expected as much since Vmware announced their abrupt licensing changes, in the same survey that I commented on last night for another reason, another site has reported on another aspect of it – nearly 40% of respondents are strongly considering moving away from Vmware in the coming year, 47% of which cite the licensing charges as the cause.
A Gartner analyst questions the numbers saying the move will be more complicated than people think and that will help Vmware retain share. I don’t agree with that myself I suspect for most customers the move will probably not be complex at all.
Myself I was just recently trying to a dig a bit more into KVM trying to figure out what they use for storage, it seems for block based systems they are using GFS2 (can’t find the link off hand)? Though I imagine they can run on top of NFS too. I wonder what the typical deployment is for KVM when it comes to storage – is shared storage widely used or is it instead used mostly with local DAS?
I just read an interesting comment from a Xen user (I’ve never found Xen to be a compelling platform myself from a technology perspective, my own personal use of Xen has been mostly indirect by means of EC2 – which in general is an absolutely terrible experience), from a thread on slashdot about this topic -
Hyper-V is about 5 years behind and XenServer is about 3 years behind in terms of functionality and stability, mainly due to the fact that VMWare has been doing it for so long. VMWare is rock-solid and feature rich, and I’d love to use them. Currently we use XenServer, but with Citrix recently closing down their hardware API’s and not playing nicely with anyone it looks like it is going to be the first casualty. I’ve been very upset by XenServer’s HA so far, plain and simple it has sucked. I’ve had hosts reboot from crashes and the virtual machines go down, but the host thinks it has the machines and all of the other hosts think it has the machines. I’ve done everything XenServer has asked (HA quorum on a separate LUN, patches, etc), but it still just sucks. I’ve yet to see a host fail and the machines to go elsewhere, and the configuration is absolutely right and has been reviewed by Citrix. Maybe 6.0 will be better, but I just heard of major issues today with it. Hyper-V is really where the competition is going to come from, especially with how engrained it is in everything coming up. Want to run Exchange 2010 SP2? Recommendation is Hyper-V virtual machines.
God I miss VMWare.
I hope Vmware comes through for me and produces a price point for the basic vSphere services that is more cost effective(basically I’d like to see vSphere Standard edition with say something crazy like 256GB/socket vRAM with the current pricing). Though I’d settle for with whatever vRAM is available in enterprise plus.
So your actually paying more for the features.
I can certainly find ways to “make do” at a cost of $1,318/socket (w/1 year of enterprise support based on this pricing), for Standard edition (includes Vmotion and HA), vs $4,369/socket for Enterprise plus. Two sockets would be around $2,600 — which is less than where vSphere 3 was, which was in the $3,000-3,500 range per pair of sockets for standard edition in 2007.
I’m not holding my breath though(since being kicked in the teeth with vSphere 5 licensing changes).
Time will tell if there are such defections, unlike Netflix where the commitment is basically zero, we’ll have to wait for the next round of hardware refreshes to kick in to see what sort of impact there is from the licensing change. Speaking of hardware refreshes(that need vSphere 5) what the hell is taking so long with the Opteron 6200s, AMD?! I really thought they’d show up in September, then couldn’t imagine them not showing up in October, and here we are at November, and still no word.
Vmware does need a “Netflix moment”, a term that has been used quite a bit recently.
Aug/117
Farewell Terremark – back to co-lo
TechOps Guy: Nate
I mentioned not long ago that I was going co-lo once again. I was co-lo for a while for my own personal services but then my server started to act up (the server was 6 years old if it was still alive today) with disk “failure” after failure (or at least that’s what the 3ware card was predicting eventually it stopped complaining and the disk never died again). So I thought – do I spent a few grand to buy a new box or go “cloud”. I knew up front cloud would cost more in the long run but I ended up going cloud anyways as a stop gap – I picked Terremark because it had the highest quality design at the time(still does).
During my time with Terremark I never had any availability issues, there was one day where there was some high latency on their 3PAR arrays though they found & fixed whatever it was pretty quick (didn’t impact me all that much).
I had one main complaint with regards to billing – they charge $0.01 per hour for each open TCP or UDP port on their system, and they have no way of doing 1:1 NAT. For a web server or something this is no big deal, but for me I needed a half dozen or more ports open per system(mail, dns, vpn, ssh etc) after cutting down on ports I might not need, so it starts to add up, indeed about 65% of my monthly bill ended up being these open TCP and UDP ports.
Once both of my systems were fully spun up (the 2nd system only recently got fully spun up as I was too lazy to move it off of co-lo) my bill was around $250/mo. My previous co-lo was around $100/mo and I think I had them throttle me to 1Mbit of traffic (this blog was never hosted at that co-lo).
The one limitation I ran into on their system was that they could not assign more than 1 IP address for outbound NAT per account. In order to run SMTP I needed each of my servers to have their own unique outbound IP. So I had to make a 2nd account to run the 2nd server. Not a big deal(for me, ended up being a pain for them since their system wasn’t setup to handle such a situation), since I only ran 2 servers (and the communications between them were minimal).
As I’ve mentioned before, the only part of the service that was truly “bill for what you use” was bandwidth usage, and for that I was charged between 10-30 cents/month for my main system and 10 cents/month for my 2nd system.
Oh – and they were more than willing to setup reverse DNS for me which was nice (and required for running a mail server IMO). I had to agree to a lengthy little contract that said I wouldn’t spam in order for them to open up port 25. Not a big deal. The IP addresses were “clean” as well, no worries about black listing.
Another nice thing to have if they would of offered it is billing based on resource pools, as usual they charge for what you provision(per VM) instead of what you use. When I talked to them about their enterprise cloud offering they charged for the resource pool (unlimited VMs in a given amount of CPU/memory), but this is not available on their vCloud Express platform.
It was great to be able to VPN to their systems to use the remote console (after I spent an hour or two determining the VPN was not going to work in Linux despite my best efforts to extract linux versions of the vmware console plugin and try to use it). Mount an ISO over the VPN and install the OS. That’s how it should be. I didn’t need the functionality but I don’t doubt I would of been able to run my own DHCP/PXE server there as well if I wanted to install additional systems in a more traditional way. Each user gets their own VLAN, and is protected by a Cisco firewall, and load balanced by a Citrix load balancer.
A couple of months ago the thought came up again of off site backups. I don’t really have much “critical” data but I felt I wanted to just back it all up, because it would be a big pain if I had to reconstruct all of my media files for example. I have about 1.7TB of data at the moment.
So I looked at various cloud systems including Terremark but it was clear pretty quick no cloud company was going to be able to offer this service in a cost effective way so I decided to go co-lo again. Rackspace was a good example they have a handy little calculator on their site. This time around I went and bought a new, more capable server.
So I went to a company I used to buy a ton of equipment from in the bay area and they hooked me up with not only a server with ESXi pre-installed on it but co-location services (with “unlimited” bandwidth), and on-site support for a good price. The on-site support is mainly because I’m using their co-location services(which in itself is a co-lo inside Hurricane Electric) and their techs visit the site frequently as-is.
My server is a single socket quad core processor, 4x2TB SAS disks (~3.6TB usable which also matches my usable disk space at home which is nice – SAS because VMware doesn’t support VMFS on SATA though technically you can do it the price premium for SAS wasn’t nearly as high as I was expecting), 3ware RAID controller with battery backed write-back cache, a little USB thing for ESXi(rather have ESXi on the HDD but 3ware is not supported for booting ESXi), 8GB Registered ECC ram and redundant power supplies. Also has decent remote management with a web UI, remote KVM access, remote media etc. For co-location I asked (and received) 5 static IPs (3 IPs for VMs, 1 IP for ESX management, 1 IP for out of band management).
My bandwidth needs are really tiny, typically 1GB/month. Though now with off site backups that may go up a bit (in bursts). Only real drawback to my system is the SAS card does not have full integration with vSphere so I have to use a cli tool to check the RAID status, at some point I’ll need to hook up nagios again and run a monitor to check on the RAID status. Normally I setup the 3Ware tools to email me when bad things happen, pretty simple, but not possible when running vSphere.
The amount of storage on this box I expect to last me a good 3-5 years. The 1.7TB includes every bit of data that I still have going back a decade or more – I’m sure there’s a couple hundred gigs at least I could outright delete because I may never need it again. But right now I’m not hurting for space so I keep it there, on line and accessible.
My current setup
- One ESX virtual switch on the internet that has two systems on it – a bridging OpenBSD firewall, and a Xangati system sniffing packets(still playing with Xangati). No IP addresses are used here.
- One ESX virtual switch for one internal network, the bridging firewall has another interface here, and my main two internet facing servers have interfaces here, my firewall has another interface here as well for management. Only public IPs are used here.
- One ESX virtual switch for another internal network for things that will never have public IP addresses associated with them, I run NAT on the firewall(on it’s 3rd/4th interfaces) for these systems to get internet access.
I have a site to site OpenVPN connection between my OpenBSD firewall at home and my OpenBSD firewall on the ESX system, which gives me the ability to directly access the back end, non routable network on the other end.
Normally I wouldn’t deploy an independent firewall, but I did in this case because, well I can. I do like OpenBSD’s pf more than iptables(which I hate), and it gives me a chance to play around more with pf, and gives me more freedom on the linux end to fire up services on ports that I don’t want exposed and not have to worry about individually firewalling them off, so it allows me to be more lazy in the long run.
I bought the server before I moved, once I got to the bay area I went and picked it up and kept it over a weekend to copy my main data set to it then took it back and they hooked it up again and I switched my systems over to it.
The server was about $2900 w/1 year of support, and co-location is about $100/mo. So disk space alone the first year(taking into account cost of the server) my cost is about $0.09 per GB per month (3.6TB), with subsequent years being $0.033 per GB per month (took a swag at the support cost for the 2nd year so that is included). That doesn’t even take into account the virtual machines themselves and the cost savings there over any cloud. And I’m giving the cloud the benefit of the doubt by not even looking at the cost of bandwidth for them just the cost of capacity. If I was using the cloud I probably wouldn’t allocate all 3.6TB up front but even if you use 1.8TB which is about what I’m using now with my VMs and stuff the cost still handily beats everyone out there.
What’s the most crazy is I lack the purchasing power of any of these clouds out there, I’m just a lone consumer, that bought one server. Granted I’m confident the vendor I bought from gave me excellent pricing due to my past relationship, though probably still not on the scale of the likes of Rackspace or Amazon and yet I can handily beat their costs without even working for it.
What surprised me most during my trips doing cost analysis of the “cloud” is how cheap enterprise storage is. I mean Terremark charges $0.25/GB per month(on SATA powered 3PAR arrays), Rackspace charges $0.15/GB per month(I believe Rackspace just uses DAS). I kind of would of expected the enterprise storage route to cost say 3-5x more, not less than 2x. When I was doing real enterprise cloud pricing storage for the solution I was looking for typically came in at 10-20% of the total cost, with 80%+ of the cost being CPU+memory. For me it’s a no brainier – I’d rather pay a bit more and have my storage on a 3PAR of course (when dealing with VM-based storage not bulk archival storage). With the average cost of my storage for 3.6TB over 2 years coming in at $0.06/GB it makes more sense to just do it myself.
I just hope my new server holds up, my last one lasted a long time, so I sort of expect this one to last a while too, it got burned in before I started using it and the load on the box is minimal, would not be too surprised if I can get 5 years out of it – how big will HDDs be in 5 years?
I will miss Terremark because of the reliability and availability features they offer, they have a great service, and now of course are owned by Verizon. I don’t need to worry about upgrading vSphere any time soon as there’s no reason to go to vSphere 5. The one thing I have been contemplating is whether or not to put my vSphere management interface behind the OpenBSD firewall(which is a VM of course on the same box). Kind of makes me miss the days of ESX 3, when it had a built in firewall.
I’m probably going to have to upgrade my cable internet at home, right now I only have 1Mbps upload which is fine for most things but if I’m doing off site backups too I need more performance. I can go as high as 5Mbps with a more costly plan. 50Meg down 5 meg up for about $125, but I might as well go all in and get 100meg down 5 meg up for $150, both plans have a 500GB cap with $0.25/GB charge for going over. Seems reasonable. I certainly don’t need that much downstream bandwidth(not even 50Mbps I’d be fine with 10Mbps), but really do need as much upstream as I can get. Another option could be driving a USB stick to the co-lo, which is about 35 miles away, I suppose that is a possibility but kind of a PITA still given the distance, though if I got one of those 128G+ flash drives it could be worth it. I’ve never tried hooking up USB storage to an ESX VM before, assuming it works? hmmmm..
Another option I have is AT&T Uverse, which I’ve read good and bad things about – but looking at their site their service is slower than what I can get through my local cable company (which truly is local, they only serve the city I am in). Another reason I didn’t go with Uverse for TV is due to the technology they are using I suspected it is not compatible with my Tivo (with cable cards). Though AT&T doesn’t mention their upstream speeds specifically I’ll contact them and try to figure that out.
I kept the motherboard/cpus/ram from my old server, my current plan is to mount it to a piece of wood and hang it on the wall as some sort of art. It has lots of colors and little things to look at, I think it looks cool at least. I’m no handyman so hopefully I can make it work. I was honestly shocked how heavy the copper(I assume) heatsinks were, wow, felt like 1.5 pounds a piece, massive.
While my old server is horribly obsolete, one thing it does have even on my new server is being able to support more ram. Old server could go up to 24GB(I had a max of 6GB at the time in it), new server tops out at 8GB (have 8GB in it). Not a big deal as I don’t need 24GB for my personal stuff but just thought it was kind of an interesting comparison.
This blog has been running on the new server for a couple of weeks now. One of these days I need to hook up some log analysis stuff to see how many dozen hits I get a month.
If Terremark could fix three areas of their vCloud express service – one being resource pool-based billing, another being relaxing the costs behind opening multiple ports in the firewall (or just giving 1:1 NAT as an option), and the last one being thin provisioning friendly billing for storage — it would really be a much more awesome service than it already is.
Aug/114
Mac Daddy P10000
TechOps Guy: Nate
It’s finally here, the HP P10000 – aka 3PAR V Class. 3PAR first revealed this to their customers more than a year ago, but the eagle has landed now.
When it comes to the hardware – bigger is better (usually means faster too)
Comparisons of recent 3PAR arrays
| Array | Raw Capacity | Fibre Ports | Data Cache | Control Cache | Disks | Interconnect Bandwidth | I/O Bandwidth | SPC-1 IOPS |
|---|---|---|---|---|---|---|---|---|
| 8-node P10000 (aka V800) | 1,600 TB | 288 ports (192 host) | 512 GB | 256 GB | 1,920 | 112 GB/sec | 96 GB/sec | 600,000 (guess) |
| 8-node T800 | 800 TB | 192 ports (128 host) | 96 GB | 32 GB | 1,280 | 45 GB/sec | 19.2 GB/sec | 225,000 |
| 4-node T800 (or 4-node T400) | 400 TB | 96 (64 host) | 48 GB | 16 GB | 640 | 9.6 GB/sec | ? | ~112,000 (estimate) |
| 4-node F400 | 384 TB | 32 (24 host) | 24 GB | 16 GB | 384 | 9.6 GB/sec ? | ? | 93,000 |
3PAR V800 ready to fight
The new system is based on their latest Generation 4 ASIC, and for the first time they are putting two ASICs in each controller. This is also the first system that supports PCI Express, with if my memory serves 9 PCI Express buses per controller. Front end throughput is expected to be up in the 15 Gigabytes/second range (up from ~6GB on the T800). Just think they have nearly eight times the interconnect bandwidth than the controllers have capacity to push data to hosts, that’s just insane.
IOPS – HP apparently is not in a big rush to post SPC-1 numbers, but given the increased spindle count, cache, doubling up on ASICs, and the new ASIC design itself I would be surprised if the system would get less than say half a million IOPS on SPC-1 (by no means a perfect benchmark but at least it’s a level playing field).
It’s nice to see 3PAR finally bulk up on data cache (beefcake!!) – I mean traditionally they don’t need it all that much because their architecture blows the competition out of the water without breaking a sweat – but still – ram is cheap – it’s not as if they’re using the same type of memory you find in CPU cache – it’s industry standard ECC DIMMs. RAM may be cheap, but I’m sure HP won’t charge you industry standard DIMM pricing when you go to put 512GB in your system!
Now that they have PCI Express 3PAR can natively support 8Gbps fibre channel as well as 10Gbit iSCSI and FCoE which are coming soon.
The drive cages and magazines are more or less unchanged (physically) from the previous generation but apparently new stuff is still coming down the pike there. The controller’s physical design (how it fits in the cabinet) seems radically different than their previous S or T series.
Another enhancement for this system is they expanded the number of drive chassis to 48, or 12 per node (up from 8 per node). Though if you go back in time you’ll find their earliest S800 actually supported 64 drive chassis for a time, since then they have refrained from daisy chaining drive chassis on their S/T/V class which is how they achieved the original 64 drive chassis configuration (or 2,560 disks back when disks were 9GB in size). The V class obviously has more ports so they can support more cages. I have no doubt they could go to even more cages by using ports assigned to hosts and assign them to disks, just a matter of testing. Flipping a fiber port from host to disk is pretty trivial on the system.
The raw capacity doesn’t quite line up with the massive amount of control cache the system has, in theory at least if 4GB of control cache per controller is good enough for 200TB raw (per controller pair), then 32GB per controller should be able to net you 1,600 TB raw (per controller pair or 6,400 TB for the whole system), but obviously with a limit put in of 1,600 TB for the entire system they are using a lot of control cache for something else.
As far as I know the T-class isn’t going anywhere anytime soon, this V class is all about even more massive scale, at a significantly higher entry level price point than the T-class(at least $100,000 more at the baseline from what I can tell), with the beauty of running the same operating system, the same user interfaces, the same software features across the entire product line. The T-class, as-is still is mind numbingly fast and efficient, even three years after it was released.
No mainframe connectivity on this baby.
Storage Federation
The storage federation stuff is pretty cool in that it is peer based, you don’t need any external appliances to move the data around, the arrays talk to each other directly to manage all of that. This is where we get the first real integration between 3PAR and HP in that the entire line of 3PAR arrays as well as the Lefthand-based P4000 iSCSI systems (including the Virtual storage appliance even!) support this new peer federation (sort of makes me wonder where EVA support is – perhaps it’s coming later or maybe it’s a sign HP is sort of depreciating EVA when it comes to this sort of thing – I’m sure the official party line will be EVA is still a shining star).
The main advantage I think of storage federation technology over something like storage vMotion is the array has a more holistic view of what’s going on in the storage system rather than just what a particular host sees, or what a particular LUN is doing. The federation should also have more information about the location of the various arrays if they are in another data center or something and make more intelligent choices about moving stuff around. Certainly would like to see it in action myself. Even though hypervisors have had thin provisioning for a while – by no means does it reduce the need for thin provisioning at the storage level (at least for larger deployments).
I’d imagine like most things on the platform the storage federation is licensed based on the capacity of the array.
If this sort of thing interests you anywhere nearly as much as it interests me you should check out the architecture white paper from HP which has some new stuff from the V class here. You don’t have to register to download it like you did back in the good ‘ol days.
I’d be surprised if I ever decided to work for a company large enough to be able to leverage a V-class, but if anyone from 3PAR is out there reading this (I’m sure there’s more than one) since I am in the Bay area – not far from your HQ – I wouldn’t turn down an invitation to see one of these in person
Oh HP.. first you kick me in the teeth by killing WebOS devices then before I know what happened you come out with a V-class and want to make things all better, I just don’t know what to feel.
The joys of working with a 3PAR array, it’s been about a year since I laid my hands on one (working at a different company now), I do miss it.
Aug/110
VMware revamps vSphere 5 licensing again
TechOps Guy: Nate
I guess someone over there high up was listening, nice to see the community had some kind of impact, VMware has adjusted their policies to some degree, far from perfect, but more bearable than the original plan.
The conspiracy theorist makes me think VMware put bogus numbers out there to begin with, never having any intension of following through with them to gauge the reaction, and then adjusted them to what they probably originally would of offered and try to make people think like they “won” by getting VMware to reduce the impact to some degree.
vSphere Enterprise List Pricing comparison (w/o support)
| # of Sockets | RAM | vSphere 4 Enterprise | vSphere 5 Enterprise (old) | vSphere 5 Enterprise (new) | Cost increase over vSphere 4 |
|---|---|---|---|---|---|
| 2 | 256GB | 2 Licenses - $5,750 | 8 Licenses - $23,000 | 4 Licenses - $11,500 | 100% |
| 4 | 512GB | N/A | 16 Licenses - $46,000 | 8 Licenses - $23,000 | N/A |
| 8 | 1024GB | N/A | 32 Licenses - $92,000 | 16 Licenses - $46,000 | N/A |
vSphere Enterprise+ List Pricing comparison (w/o support)
| # of Sockets | RAM | vSphere 4 Enterprise+ | vSphere 5 Enterprise+ (old) | vSphere 5 Enterprise+ (new) | Cost increase over vSphere 4 |
|---|---|---|---|---|---|
| 2 | 256GB | 2 Licenses - $6,990 | 5 Licenses (240GB) - $17,475 | 3 Licenses (288GB) - $10,485 | 50% higher |
| 4 | 512GB | 4 Licenses - $13,980 | 11 Licenses (528GB) - $38,445 | 5 Licenses (480GB) - $17,475 | 25% higher |
| 8 | 1024GB | 8 Licenses - $27,960 | 21 Licenses(1008GB) - $73,995 | 11 Licenses (1056GB) - $38,445 | 37% higher |
There were other changes too, see the official VMware blog post above for the details. They quadrupled the amount of vRAM available for the free ESXi to 32GB which I still think is not enough, should be, say at least 128GB.
Also of course they are pooling their licenses so the numbers fudge out a bit more depending on the # of hosts and stuff.
One of the bigger changes is VMs larger than 96GB will not need more than 1 license. Though I can’t imagine there are many 96GB VMs out there… even with 1 license if I wanted several hundred gigs of ram for a system I would put in on real hardware, get more cpu cores to boot (not unlikely you have 48-64+ cores of cpu for such a system, which is far beyond where vSphere 5 can scale to for a single VM).
I did some rounding in the price estimates, because the numbers are not divisible cleanly by the amount of ram specified.
It seems VMware has effectively priced their “Enterprise” product out of the market if you have any more than a trivial amount of memory. vSphere 4 Enterprise was, of course limited to 256GB of ram, but look at the cost of that compared to the new stuff, pretty staggering.
Quad socket 512GB looks like the best bet on these configurations anyways.
I still would like to see pricing based more on features than on hardware. E.g. give me vSphere standard edition with 96GB per CPU of vRAM licensing, because a lot of those things in Enteprise+ I don’t need (some are nice to have but very few are critical for most people I believe). As-is users are forced into the higher tiers due to the arbitrary limits set on the licensing, not as bad as the original vSphere 5 pricing but still pretty bad for some users when compared to vSphere 4.
Or give me free ESXi with the ability to individually license software features such as vMotion etc on top of it on a per-socket basis or something.
I think the licensing scheme needs more work. VMware could also do their customers a favor by communicating how this will change in the future, as bigger and bigger machines come out it’s logical to think the memory limits would be increased over time.
The biggest flaw in the licensing scheme remains it measures based on what is provisioned, rather than what is used. There is no excuse for this from VMware since they own the hypervisor and have all the data.
Billing based on provision vs usage is the biggest scam in this whole cloud era.
Jul/110
VMware Licensing models
TechOps Guy: Nate
[ was originally combined with another post but I decided to split out ]
VMware has provided it’s own analysis of their customers hardware deployments and telling folks that ~95% of their customers won’t be impacted by the licensing changes. I feel pretty confident that most of those customers are likely massively under utilizing their hardware. I feel confident because I went through that phase as well. Very, very few workloads are truly cpu bound especially with 8-16+ cores per socket.
It wouldn’t surprise me at all that many of those customers when they go to refresh their hardware change their strategy pretty dramatically – provided the licensing permits it. The new licensing makes me think we should bring back 4GB memory sticks and 1 GbE. It is very wasteful to assign 11 CPU licenses to a quad socket system with 512GB of memory, memory only licenses should be available at a significant discount over CPU+memory licenses at the absolute minimum. Not only that but large amounts of memory are actually affordable now. It’s hard for me to imagine at least having a machine with a TB of memory in it for around $100k, it wasn’t TOO long ago that it would of run you 10 times that.
And as to VMware’s own claims that this new scheme will help align ANYTHING better, by using memory pools across the cluster – just keep this in mind. Before this change we didn’t have to care about memory at all, whether we used 1% or 95%, whether some hosts used all of their ram and others used hardly any. It didn’t matter. VMware is not making anything simpler. I read somewhere about them saying some crap about aligning more with IT as a service. Are you kidding me? How may buzz words do we need here?
The least VMware can do is license based on usage. Remember pay for what you use, not what you provision. When I say usage I mean actual usage. Not charging me for the memory my Linux systems are allocating towards (frequently) empty disk buffers (goes to the memory balloon argument). If I allocate 32GB of ram to a VM that is only using 1GB of memory I should be charged for 1GB, not 32GB. Using vSphere’s own active memory monitor would be an OK start.
Want to align better and be more dynamic? align based on memory usage and CPU usage, let me run unlimited cores on the cluster and you can monitor actual usage on a per-socket basis, so if on average (say you can bill based on 95% similar to bandwidth) your using 40% of your CPU then you only need 40% licensing. I still much prefer the flat licensing model in almost any arrangement rather than usage based but if your going to make it usage based, really make it usage based.
Oh yeah – and forget about anything that charges you per VM too (hello SRM). That’s another bogus licensing scheme. It goes completely against the trend of splitting workloads up into more isolated VMs and instead favors fewer much larger VMs that are doing a lot of things at the same time. Even on my own personal co-located ESXi server, I have 5 VMs on it, I could consolidate it to two and provide the similar end user services, but it’s much cleaner to do it in 5 for my own sanity.
All of this new licensing stuff also makes me think back to a project I was working on about a year ago, trying to find some way of doing DR in the cloud, the ROI for doing it in house vs. any cloud on the market(looked at about 5 different ones at the time) was never more than 3 months. In one case the up front costs for the cloud was 4 times the cost for doing it internally. The hardware needs were modest in my opinion, with the physical hardware not even requiring two full racks of equipment. The #1 cost driver was memory, #2 was CPU, storage was a distant third assuming the storage that the providers spec’d could meet the IOPS and throughput requirements, storage came in at about 10-15% of the total cost of the cloud solution.
Since most of my VMware deployments have been in performance sensitive situations (lots of Java) I run the systems with zero swapping, everything in memory has to stay in physical ram.
Jul/110
Cluster DRS
TechOps Guy: Nate
Given the recent price hikes that VMware is imposing on it’s customers(because they aren’t making enough money obviously) , and looking at the list of new things in vSphere 5 and being, well underwhelmed (compared to vSphere 4), I brain stormed a bit and thought about what kind of things I’d like to see VMware add.
VMware seems to be getting more aggressive in going after service providers (their early attempts haven’t been successful, it seems they have less partners now than a year ago – btw I am a vCloud express end-user at the moment). An area that VMware has always struggled in is scalability in their clusters (granted such figures have not been released for vSphere 5 but I am not holding my breath for a 10-100x+ increase in scale)
Whether it’s the number of virtual machines in a cluster, the number of nodes, the scalability of the VMFS file system itself (assuming that’s what your using) etc.
For the most part of course, a cluster is like a management domain, which means it is, in a way a single point of failure. So it’s pretty common for people to build multiple clusters when they have a decent number of systems, if someone has 32 servers, it is unlikely they are going to build a single 32-node cluster.
A feature I would like to see is Cluster DRS, and Cluster HA. Say for example you have several clusters, some clusters are very memory heavy for loading a couple hundred VMs/host(typically 4-8 socket with several hundred gigs of ram), others are compute heavy with very low cpu consolidation ratios (probably dual socket with 128GB or less of memory). Each cluster by itself is a stand alone cluster, but there is loose logic that binds them together to allow the seamless transport of VMs between clusters either for either load balancing or fault tolerance. Combine and extend regular DRS to span clusters, on top of that you may need to do transparent storage vMotion (if required) as well along with the possibility of mapping storage on the target host (on the fly) in order to move the VM over (the forthcoming storage federation technologies could really help make hypervisor life simpler here I think).
Maybe a lot of this could be done using yet another management cluster of some kind, a sort of independent proxy of things (running on independent hardware and perhaps even dedicated storage). In the unlikely event of a catastrophic cluster failure, the management cluster would pick up on this and move the VMs to other clusters and re start them (provided there is sufficient resources of course!). In very large environments it is not be possible to map everything to everywhere, which would require multiple storage vMotions in order to get the VM from the source to a destination that the target host can access – if this can be done at the storage layer via the block level replication stuff first introduced in VAAI that could of course greatly speed up what otherwise might be a lengthy process.
Since it is unlikely anyone is going to be able to build a single cluster with shared storage that spans a great many systems(100s+) and have it be bulletproof enough to provide 99.999% uptime, this kind of capability would be a stop gap, providing the flexibility and availability of a single massive cluster, while at the same time reducing the complexity in having to try to build software that can actually pull the impossible (or what seems impossible today) off.
On the topic of automated cross cluster migrations, having global spare hardware would be nice too, much like most storage arrays have global hot spares, which can be assigned to any degraded RAID group on the system regardless of what shelf it may reside on. Global spare servers would be shared across clusters, and assigned on demand. A high end VM host is likely to cost upwards of $50,000+ in hardware these days, multiply by X number of clusters and well.. you get the idea.
While I’m here, I might as well say I’d like the ability to hot remove memory, Hyper-V has dynamic memory which seems to provide this functionality. I’m sure the guest OSs would need to be re-worked a bit too in order to support this, since in the physical world it’s not too common to need to yank live memory from a system. In the virtual world it can be very handy.
Oh and I won’t forget – give us an ability to manually control the memory balloon.
Another area that could use some improvement is the vMotion compatibility, there is EVC, but last I read you still couldn’t cross processor manufacturers when doing vMotion with EVC. KVM can apparently do it today.
Jul/117
VMware jacks up prices too
TechOps Guy: Nate
Not exactly hot on the heels of Red Hat’s 260% price increase, VMware has done something similar with the introduction of vSphere 5 which is due later this year.
The good: They seem to have eliminated the # of core/socket limit for each of the versions, and have raised the limit of vCPUs per guest to 8 from 4 on the low end, and to 32 from 8 on the high end.
The bad: They have tied licensing to the amount of memory on the server. Each CPU license is granted a set amount of memory it can address.
The ugly: The amount of memory addressable per CPU license is really low.
Example 1 – 4x[8-12] core CPUs with 512GB memory
- vSphere 4 cost with Enterprise Plus w/o support (list pricing) = ~$12,800
- vSphere 5 cost with Enterprise Plus w/o support (list pricing) = ~$38,445
- vSphere 5 cost with Enterprise w/o support (list pricing) = ~$46,000
- vSphere 5 cost with Standard w/o support (list pricing) = ~$21,890
So you pay almost double for the low end version of vSphere 5 vs the highest end version of vSphere 4.
Yes you read that right, vSphere 5 Enterprise costs more than Enterprise Plus in this example.
Example 2 – 8×10 core CPUs with 1024GB memory
- vSphere 4 cost with Enterprise Plus w/o support (list pricing) = ~$25,600
- vSphere 5 cost with Enterprise Plus w/o support (list pricing) = ~$76,890
It really is an unfortunate situation, while it is quite common to charge per CPU socket, or in some cases per CPU core, I have not heard of a licensing scheme that charged for the memory.
I have been saying that I would expect to be using VMware vSphere myself until the 2012 time frame at which point I hope KVM is mature enough to be a suitable replacement (I realize there are some folks out there using KVM now it’s just not mature enough for my own personal taste).
The good news, if you can call it that, is as far as I can tell you can still buy vSphere 4 licenses, and you can even convert vSphere 5 licenses to vSphere 4 (or 3). Hopefully VMware will keep the vSphere 4 license costs around for the life of (vSphere 4) product, which would take customers to roughly 2015.
I have not seen much info about what is new in vSphere 5, for the most part all I see are scalability enhancements for the ultra high end (e.g. 36Gbit/s network throughput, 1 million IOPS, supporting more vCPUs per VM – number of customers that need that I can probably count on 1 hand). With vSphere 4 there was many good technological improvements that made it compelling for pretty much any customer to upgrade (unless you were using RDM with SAN snapshots), I don’t see the same in vSphere 5 (at least at the core hypervisor level). My own personal favorites for vSphere 4 enhancements over 3 were – ESXi boot from SAN, Round Robin MPIO, and the significant improvements in the base hypervisor code itself.
I can’t think of a whole lot of things I would want to see in vSphere 5 that aren’t already in vSphere 4, my needs are somewhat limited though. Most of the features in vSphere 4 are nice to have though for my own needs are not requirements. For the most part I’d be happy on vSphere standard edition (with vMotion which was added to the licensed list for Standard edition about a year ago) the only reason I go for higher end versions is because of license limitations on hardware. The base hypervisor has to be solid as a rock though.
In my humble opinion, the memory limits should look more like
- Standard = 48GB (Currently 24GB)
- Enterprise = 96GB (Currently 32GB)
- Enterprise Plus = 128GB (Currently 48GB)
It just seems wrong to have to load 22 CPU licenses of vSphere on a host with 8 CPUs and 1TB of memory.
I remember upgrading from ESX 3.5 to 4.0, it was so nice to see that it was a free upgrade for those with current support contracts.
I have been a very happy, loyal and satisfied user & customer of VMware’s products since 1999, put simply they have created some of the most robust software I have ever used (second perhaps to Oracle). Maybe I have just been lucky over the years but the number of real problems (e.g. caused downtime) I have had with their products has been tiny, I don’t think it’s enough to need more than one hand to count. I have never once had a ESX or GSX server crash for example. I see mentions of the PSOD that ESX belches out on occasion but I have yet to see it in person myself.
I’ve really been impressed by the quality and performance (even going back as far as my first e-commerce launch on VMware GSX 3.0 in 2004 we did more transactions the first day than we were expecting for the entire first month), so I’m happy to admit I have become loyal to them over the years(for good reason IMO). Pricing moves like this though are very painful, and it will be difficult to break that addiction.
This also probably means if you want to use the upcoming Opteron 6200 16-core cpus (also due in Q3) on vSphere you probably have to use vSphere 5, since 4 is restricted to 12-cores per socket (though would be interesting to see what would happen if you tried).
If I’m wrong about this math please let me know, I am going by what I read here.
Microsoft’s gonna have a field day with these changes.
And people say there’s no inflation going on out there..
sigh
Jan/116
Terremark snatched by Verizon
TechOps Guy: Nate
Sorry for my three readers out there for not posting recently I’ve been pretty busy! And to me there hasn’t been too much events in the tech world in the past month or so that have gotten me interested enough to write about them.
One recent event that did was Verizon’s acquisition of Terremark, a service I started using about a year ago.
I was talking with a friend of mine recently he was thinking about either throwing a 1U server in a local co-location or play around with one of the cloud service providers. Since I am doing both still (been too lazy to completely move out of the co-lo…) I gave him my own thoughts, and it sort of made me think about more about the cloud in general.
What do I expect from a cloud?
When I’m talking cloud I’m mainly referring to the IaaS or Infrastructure as a Service. Setting aside cost modelling and stuff for a moment here I expect the IaaS to more or less just work. I don’t want to have to care about:
- Power supply failure
- Server failure
- Disk drive failure
- Disk controller failure
- Scheduled maintenance (e.g. host server upgrades either software or hardware, or fixes etc)
- Network failure
- UPS failure
- Generator failure
- Dare I say it ? A fire in the data center?
- And I absolutely want to be able to run what ever operating system I want, and manage it the same way I would manage it if it was sitting on a table in my room or office. That means boot from an ISO image and install like I would anything else.
Hosting it yourself
I’ve been running my own servers for my own personal use since the mid 90s. I like the level of control it gives me and the amount of flexibility I have with running my own stuff. Also gives me a playground on the internet where I can do things. After multiple power outages over the first part of the decade, one of which lasted 28 hours, and the acquisition of my DSL provider for the ~5th time, I decided to go co-lo. I already had a server and I put it in a local, Tier 2 or Tier 3 data center. I could not find a local Tier 4 data center that would lease me 1U of space. So I lacked:
- Redundant Power
- Redundant Cooling
- Redundant Network
- Redundant Servers (if my server chokes hard I’m looking at days to a week+ of downtime here)
For the most part I guess I had been lucky, the facility had one, maybe two outages since I moved in about three years ago. The bigger issue with my server was aging and the disks were failing, it was a pain to replace them and it wasn’t going to be cheap to replace the system with something modern and capable of running ESXi in a supported configuration(my estimates put the cost at a minimum of $4k). Add to that the fact that I need such a tiny amount of server resources.
Doing it right
So I had heard of Terremark from my friends over at 3PAR, and you know I like 3PAR, and they use Vmware and I like Vmware. So I decided to go with them rather than the other providers out there, they had a decent user interface and I got up and going fairly quickly.
So I’ve been running it for almost a year, with pretty much no issues, I wish they had a bit more flexibility in the way they provision networking stuff but nothing is perfect (well unless you have the ability to do it yourself).
From a design perspective, Terremark has done it right, whether it’s providing an easy to use interface to provision systems, using advanced technology such as VMware, 3PAR, and Netscaler load balancers, and building their data centers to be even — fire proof.
Having the ability to do things like Vmotion, or Storage vMotion is just absolutely critical for a service provider, I can’t imagine anyone being able to run a cloud without such functionality at least with a diverse set of customers. Having things like 3PAR’s persistent cache is critical as well to keep performance up in the event of planned or unplanned downtime in the storage controllers.
I look forward to the day where the level of instrumentation and reporting in the hypervisors allow billing based on actual usage, rather than what is being provisioned up front.
Sample capabilities
In case your a less technical user I wanted to outline a few of the abilities the technology Terremark uses offers their customers -
Memory Chip Failure (or any server component failure or change)
Most modern servers have sensors on them and for the most part are able to accurately predict when a memory chip is behaving badly and to warn the operator of the machine to replace it. But unless your running on some very high end specialized equipment (which I assume Terremark is not because it would cost too much for their customers to bare), the operator needs to take the system off line in order to replace the bad hardware. So what do they do? They tell VMware to move all of the customer virtual machines off the affected server onto other servers, this is done without customer impact, the customer never knows this is going on. The operator can then take the machine off line and replace the faulty components and then reverse the process.
Same applies to if you need to:
- Perform firmware or BIOS updates/changes
- Perform Hypervisor updates/patches
- Maybe your retiring an older type of server and moving to a more modern system
Disk failure
This one is pretty simple, a disk fails in the storage system and the vendor is dispatched to replace it, usually within four hours. But they may opt to wait a longer period of time for whatever reason, with 3PAR it doesn’t really matter, there are no dedicated hot spares so your really in no danger of losing redundancy, the system rebuilds quickly using a many:many RAID relationship, and is fully redundant once again in a matter of hours(vs days with older systems and whole-disk-based RAID).
Storage controller software upgrade
There are fairly routine software upgrades on modern storage systems, the software feature set seems to just grow and grow. So the ability to perform the upgrade without disrupting users for too long(maybe a few seconds) is really important with a diverse set of customers, because there will probably be no good time where all customers say ok I have have some downtime. So having high availability storage with the ability to maintain performance with a controller being off line by mirroring the cache elsewhere is a very useful feature to have.
Storage system upgrade (add capacity)
Being able to add capacity without disruption and dynamically re-distribute all existing user data across all new as well as current disk resources on-line to maximize performance is a boon for customers as well.
UPS failure (or power strip/PDU failure)
Unlike the small dinky UPS you may have in your house or office UPSs in data centers typically are powering up to several hundred machines, so if it fails then you may be in for some trouble. But with redundant power you have little to worry about, the other power supply takes over without interruption.
If a server power supply blows up it has the ability to take out the entire branch or even whole circuit that it’s connected to. But once again redundant power saves the day.
Uh-oh I screwed up the network configuration!
Well now you’ve done it, you hosed the network (or maybe for some reason your system just dropped off the network maybe flakey network driver or something) and you can’t connect to your system via SSH or RDP or whatever you were using. Fear not, establish a VPN to the Terremark servers and you can get console access to your system. If only the console worked from Firefox on Linux..can’t have everything I guess. Maybe they will introduce support for vSphere 4.1′s virtual serial concentrators soon.
It just works
There are some applications out there that don’t need the level of reliability that the infrastructure Terremark uses can provide and they prefer to distribute things over many machines or many data centers or something, that’s fine too, but most apps, almost all apps in fact make the same common assumption, perhaps you can call it the lazy assumption – they assume that it will just work. Which shouldn’t surprise many, because achieving that level of reliability at the application layer alone is an incredibly complex task to pull off. So instead you have multiple layers of reliability under the application handling a subset of availability, layers that have been evolving for years or decades even in some cases.
Terremark just works. I’m sure there are other cloud service providers out there that work too, I haven’t used them all by any stretch(nor am I seeking them for that matter).
Public clouds make sense, as I’ve talked about in the past for a subset of functionality, they have a very long ways to go in order to replace what you can build yourself in a private cloud (assuming anyone ever gets there). For my own use case, this solution works.
Follow Nate
