TechOpsGuys.com Diggin' technology every day

November 15, 2011

LSI quietly smothers Onstor

Filed under: Storage — Nate @ 8:35 pm

About a year ago I was in the market for a new NAS box to hook up to my 3PAR T400, something to eventually replace the Exanet cluster that was hooked up to it since Exanet as a company went bust.

There wasn’t many options left, Onstor had been bought by LSI, I really couldn’t find anything on the Ibrix offering from HP at the time (at least for a SAN-attached Ibrix rather than a scale-out Ibrix), and then there was of course NetApp V-Series. I could not find any trace of Polyserve which HP acquired as well, other than something related to SQL server.

Old 3PAR graphic that I dug up from the Internet Archive

3PAR was suggesting I go with Onstor(this was, of course before the HP/Dell bidding war), claiming they still had a good relationship with Onstor through LSI. I think it was less about the partnership and more about NetApp using the V-series to get their foot in the door and then try to replace the back end disk with their own, a situation understandably 3PAR (or any other competition) doesn’t like to be in.

My VAR on the other hand had another story to tell, after trying to reach out to LSI/Onstor they determined that Onstor was basically on their death bed, with only a single reseller in the country authorized to sell the boxes, and it seemed like there was maybe a half dozen employees left working on the product.

So, I went with NetApp, then promptly left the company and left things in the hands of the rest of the team(there’s been greater than 100% turnover since I left both in the team and in the management).

One of my other friends who used to work for Exanet was suggesting to me that LSI bought Onstor with the possible intention of integrating the NAS technology into their existing storage systems, to be able to offer a converged storage option to the customers, and that the stand alone gateway would probably be going away.

Another product I had my eyes on at the time and 3PAR was working hard with to integrate was the Symantec Filestore product. I was looking forward to using it, other companies were looking to Filestore to replace their Exanet clusters as well. Though I got word through unofficial channels that Symantec planned to kill the software-only version and go the appliance route. It took longer than I was expecting but they finally did it, I was on their site recently and noticed that the only way to get it now is with integrated storage from their Chinese partner.

I kept tabs on Onstor now and then, wondering if it would come back to life in some form, the current state of the product at least from a SAN connectivity perspective seemed to be very poor – as in you couldn’t do things like live software upgrades on a cluster in most cases, the system had to have a full outage(in a lot of cases anyways). But no obvious updates ever came.

Then LSI sold their high end storage division to NetApp. I suppose that was probably the end of the road for Onstor.

So tonight, I was hitting some random sites and decided to check in on Onstor again, only to find most traces of the product erased from LSI’s site.

The only things I ever really heard about Onstor was how the likes of BlueArc and Exanet were replacing Onstor clusters, I talked to one service provider who had an Onstor system (I think connected to 3PAR too), I talked with them briefly while I was thinking about what NAS gateway to move to about a year ago and they seemed fairly content with it, no major complaints. Though it seemed like if they were to buy something new (at that time) they probably wouldn’t buy Onstor due to the uncertainty around it.

It seemed to be an interesting design – using dual processor quad core MIPS CPUs of all things.

RIP Onstor, another one bites the dust.

Hopefully LSI doesn’t do the same to 3ware. I always wondered why 3ware technology was never integrated(as far as I know anyways) into server motherboards, even after LSI acquired them, given that a lot of integrated RAID controllers (Dell etc) are LSI. I think for the most part the 3ware technology is better (if not why did they get acquired and continue to develop products?). I’ve been a 3ware user for what seems like 12 years now, and really have no complaints.

I really hope the HP X9000 NAS gateway works out, the entry level pricing for it as-is seems quite high to me though.

Dell’s distributed core

Filed under: Networking — Tags: — Nate @ 9:59 am

Dell’s Force10 unit must be feeling the heat from the competition. I came across this report which the industry body Tolly did on behalf of Dell/Force10.

Normally I think Tolly reports are halfway decent although they are usually heavily biased towards the sponsor (not surprisingly). This one though felt light on details. It felt like they rushed this to market.

Basically what Force10 is talking about is a distributed core architecture with their 32-port 40GbE Z9000 switches as what they call the spine(though sometimes they are used as the leaf), and their 48-port 10 GbE S4810 switches as what they call the leaf (though sometimes they are used as the spine).

They present 3 design options:

Force10 Distributed Core Design

I find three things interesting about these options they propose:

  • The minimum node count for spine is 4 nodes
  • They don’t propose an entirely non blocking fabric until you get to “large”
  • The “large” design is composed entirely of Z9000s, yet they keep the same spine/leaf configuration, whats keeping them from being entirely spine?

The distributed design is very interesting, though it would be a conceptual hurdle I’d have a hard time getting over if I was in the market for this sort of setup. It’s nothing against Force10 specifically I just feel safer with a less complex design (I mentioned before I’m not a fan of stacking for this same reason), less things talking to each other in such a tightly integrated fashion.

That aside though a couple other issues I have with the report is while they do provide the configuration of the switches (that IOS-like interface makes me want to stab my eyes with an ice pick) – I’m by no means familiar with Force10 configuration and they don’t talk about how the devices are managed. Are the spine switches all stacked together? Are the spine and leaf switches stacked together? Are they using something along the lines of Brocade’s VCS technology? Are the devices managed independently and they are relying on other protocols like MLAG? The web site mentions using TRILL at layer 2, which would be similar to Brocade.

The other issue I have with the report is the lack of power information, specifically would be interested (slightly, in the grand scheme of things I really don’t think this matters all that much) in the power per usable port (ports that aren’t being used for up links or cross connects). They do rightly point out that power usage can vary depending on the workload and so it would be nice to get power usage based on the same workload. Though conversely it may not matters as much, looking at the specs for the Extreme X670V (48x10GbE + 4x40GbE) says there is only 8 watts of difference between (that particular switch) 30% traffic load and 100% traffic load, seems like a trivial amount.

Extreme Networks X670V Power Usage

As far as I know the Force10 S4810 switch uses the same Broadcom chipset as the X670V.

On their web site they have a nifty little calculator where you input your switch fabric capacity and it spits out power/space/unit numbers. The numbers there don’t sound as impressive:

  • 10Tbps fabric = 9.6kW / 12 systems / 24RU
  • 15Tbps fabric = 14.4kW / 18 systems / 36RU
  • 20Tbps fabric = 19.2kW / 24 systems / 48RU

The aforementioned many times Black Diamond X-Series comes in at somewhere around 4kW (well if you want to be really conservative you could say 6.1kW assuming 8.1W/port which their report was likely high considering system configuration) and a single system to get up to 20Tbps of fabric(you could perhaps technically say it is has 15Tbps of fabric since the last 5Tbps is there for redundancy, 192 x 80Gbps = 1.5Tbps). 14.5RU worth of rack space too.

Dell claims non-blocking scalability up to 160Tbps, which is certainly a lot! Though I’m not sure what it would take for me to make the leap into a distributed system such as TRILL. Given TRILL is a layer 2 only protocol (which I complained about a while ago), I wonder how they handle layer 3 traffic, is it distributed in a similar manor? What is the performance at layer 3? Honestly I haven’t read much on TRILL at this point (mainly because it hasn’t really interested me yet), but one thing that is not clear to me(maybe someone can clarify), is is TRILL just a traffic management protocol or does it also include more transparent system management(e.g. manage multiple devices as one), or does that system management part require more secret sauce by the manufacturer.

My own, biased(of course), thoughts on this architecture, while innovative:

  • Uses a lot of power / consumes a lot of space
  • Lots of devices to manage
  • Lots of connections – complicated physical network
  • Worries over resiliency of TRILL (or any tightly integrated distributed design – getting this stuff right is not easy)
  • On paper at least seems to be very scalable
  • The Z9000 32-port 40GbE switch certainly seems to be a nice product from a pure hardware/throughput/formfactor perspective. I just came across Arista’s new 1U 40GbE switch and I think I’d prefer the Force10 design with twice the size and twice the ports purely for more line rate ports in the unit.

It would be interesting to read a bit more in depth about this architecture.

I wonder if this is going to be Force10s approach going forward, the distributed design, or if they are going to continue to offer more traditional chassis products for customers who prefer that type of setup. In theory it should be pretty easy to do both.

November 14, 2011

NetApp challenge falls without winners

Filed under: Storage — Tags: — Nate @ 9:52 am

This is just too funny.

Nobody won a million pounds of kit from NetApp because data centre nerds thought the offer was unbelievable or couldn’t be bothered with the paperwork.

NetApp UK offered an award of up to £1m in NetApp hardware, software, and services to a lucky customer that managed to reduce its storage usage by 50 per cent through using NetApp gear. The competition’s rules are still available on NetApp’s website.

For years, I have been bombarded with marketing from NetApp (indirectly from the VARs and consultants they have bribed over the years) about the efficiency of the NetApp platform, especially de-dupe, how it will save you tons of money on storage etc.

Indirectly because I absolutely refused to talk directly to the NetApp team in Seattle after how badly they treated me when I was interested in being a customer a few years ago. It seemed just as bad as EMC was (at the time). I want to be treated as a customer, not a number.

Ironically enough it was NetApp that drove me into the arms of 3PAR, long before I really understood what the 3PAR technology was all about. It was NetApp’s refusal to lend me an evaluation system for any length of time which is what sealed the deal for my first 3PAR purchase. Naturally, at the time 3PAR was falling head over heels at the opportunity to give us an eval, and so we did evaluate their product, an E200 at the time. Which in the end directly led to 4 more array purchases(with a 5th coming soon I believe), and who knows how many more indirectly as a result of my advocacy either here or as a customer reference.

Fortunately my boss at the time, was kind of like me, when the end of the road came for NetApp – and when they dropped their pants (prices) so low that most people could not ignore, my boss was to the point where NetApp could of given us the stuff for free and he would of still bought 3PAR. We asked for eval for weeks and they refused us every time until the last minute.

I have no doubt that de-dupe is effective, how effective is dependent on a large number of factors. Suffice to say I don’t buy it yet myself, at least not as a primary reason to purchase primary storage for online applications.

Anyways, I think this contest, more than anything else is a perfect example. You would think that the NetApp folks out there would of jumped on this, but there are no winners, that is too bad, but very amusing.

I can’t stop giggling.

I guess I should thank NetApp for pointing me in the right direction in the beginning of my serious foray into the storage realm.

So, thanks NetApp! 🙂

Four posts in one morning! And it’s not even 9AM yet! You’d think I have been up since 5AM writing and you’d be right.

AMD Launches Opteron 6200s

Filed under: General — Tags: , , — Nate @ 9:06 am

UPDATED I have three words:

About damn time.

I’ve been waiting for a long time for these, was expecting them months ago, had to put in orders with Opteron 6100s a few weeks ago because I couldn’t wait any longer for the 6200s. Sigh. I’m half hoping I can get HP to exchange my 6100s for 6200s since the 6100s are still sitting in boxes. Though that may be being too hopeful given my time line for deployment. One thing’s for sure though, if HP can pull it off they’ll make my decision on which version of vSphere to go with pretty easy since vSphere 4 tops out at 12 cores.

AMD has finally launched the 6200, which everyone knows is the world’s first 16-core x86-64 processor, and is socket compatible with the 6100 processor which launched over a year ago providing an easy upgrade path.

I’m just running through some of the new stuff now, one feature which is nice and I believe I mentioned it a while ago is the TDP Cap, which allows a user to set the maximum power usage of the processor, basically more granular control than technologies that were used previous to it. I don’t believe it has the ability to dynamically turn cores on and off based on this value though which is unfortunate – maybe next time. Excluding the new turbo core support which is different technology.

AMD Turbo Core

I thought this was pretty cool, I was just reading about it in their slide deck. I thought, at first it was going to be similar to the Intel Turbo or IBM Turbo technology where, if I recall right (don’t quote me), the system can more or less shut off all the other cores on the socket and turbo charge a single core to super sonic speeds. AMD Turbo core operates on all cores simultaneously by between 300-500Mhz if the workload fits the power envelope of the processor. It can do the same for half of the on board cores but instead of 300-500Mhz boost the frequency by up to 1Ghz.

Memory Enhancements

It also supports higher performance memory as well as something called LR-DIMMs, which I had never heard of before. Load Reduced DIMMs seem they allow you to add more memory to the system. Even after reading the stuff on Micron’s site I’m not sure of the advantage.

I recall on the 6100 there was a memory performance hit when you utilized all 12 memory slots per CPU socket (vs using only 8/socket). I don’t see whether this is different on the 6200 or not.

Power and Performance

The highest end, lowest power Opteron 6100 seems to be the 6176 (not to be confused with the 6176 SE). The 6176 (by itself) is not even mentioned on AMD’s site (though it is on HP’s site and my recent servers have it). It is a 2.3Ghz 12-core 80W (115W TDP) processor. It seems AMD has changed their power ratings from the ACP they were using before to the TDP (what Intel uses). If I recall right ACP was something like average processor power usage, vs TDP is peak usage(?).

The 6276 is the new high end lower power option, which is a 16-core 2.3Ghz processor with the same power usage. So they managed to squeeze in an extra 9.2Ghz worth of processing power in the same power envelope. That’s pretty impressive.

There’s not a lot of performance metrics out at this stage, but here’s something I found on AMD’s site:

SPEC Int rate_base2006 Mainstream CPUs

That’s a very good price/performance ratio. This graph is for “mainstream CPUs” that is CPUs with “normal” power usage, not ultra high end CPUs which consume a lot more power. Those are four socket systems so for the CPUs alone on the high end from Intel would run $8,236, and from AMD $3,152. Then there is the motherboard+chipset from Intel which will carry a premium over AMD as well since Intel has different price/scalability bands for their processors between their two socket and four socket systems (where AMD does not, though with Intel you can now get two socket versions of servers with the latest Intel processors they still seem to carry a decent premium since I believe they use the same chipsets as the four socket boxes the two socket versions are made more for memory capacity bound workloads rather than CPU bound).

They have floating point performance too though for the stuff I do floating point doesn’t really matter, more useful probably for SGI and Cray and their super computers.

It’s not the 3.5Ghz that AMD was talking about but I trust that is coming..at some point. AMD has been having some manufacturing issues recently which probably was the main cause for the delays of the 6200, hopefully they get those worked out in short order.

HP has already updated their stuff to reflect support for the latest processors in their existing platforms.

From HP’s site, here are the newest 16 core processors:

  • 6282SE (2.6GHz/16-core/16MB/140W TDP) Processor
  • 6276 (2.3GHz/16-core/16MB/115W TDP) Processor
  • 6274 (2.2GHz/16-core/16MB/115W TDP) Processor
  • 6272 (2.1GHz/16-core/16MB/115W TDP) Processor
  • 6262HE (1.6GHz/16-core/16MB/85W TDP) Processor

Few more stats –

  • L1 CPU Cache slashed from 128kB to 48kB (total 1,536kB to 768kB)
  • L2 CPU Cache increased from 512kB to 1,000 kB (total 6,144kB to 12,000kB)
  • L3 CPU Cache increased from 12,288 kB to 16,384 kB (1,024kB per core for both procs)
  • Memory controller clock speed increased from 1.8Ghz to 2Ghz
  • CMOS process shrunk from 45nm to 32nm

Interesting how they shifted focus away from the L1 cache to the L2 cache.

Anyone know how many transistors are on this thing? And how many were on the 6100 ? How about on some of the recent Intel chips?

Now to go figure out how much these things actually cost and what the lead times are.

UPDATE – I know pricing at least now, the new 16 core procs are, as the above graph implies actually cheaper than the 12-core versions! That’s just insane, how often does that happen?!?!

Bottom line

With so many things driving virtualization these days, and with such high consolidation ratios, especially with workloads that are not CPU constrained(which are most), myself I like the value that the 6000-series AMD chips give, especially the number of raw cores without hyperthreading. The AMD 6000 platform is the first AMD platform I have really, truly liked I want to say going back a long, long ways. I’ll admit I was mistaken in my ways for a few years when I was on the Intel bandwagon. Though I have been on the ‘give me more cores’ bandwagon ever since the first Intel quad core processor. Now that AMD has the most cores, on a highly efficient platform, I suppose I gravitate towards them now. There are limits to how far you go to get cores of course, I’m not sure what my limit is. I’ve mentioned in the past I wouldn’t be interested in something like a 48x200Mhz CPU for example. The Opteron 6000 has a nice balance of per-core performance (certainly can’t match Intel’s per core performance but it’s halfway decent especially given the price), and many, many cores.

Three blog posts in one morning, busy morning!

Oracle throws in Xen virtualization towel?

Filed under: Virtualization — Tags: , — Nate @ 7:03 am

This just hit me a few seconds ago and it gave me something else to write about so here goes.

Oracle recently released Solaris 11, the first major rev to Solaris in many many years. I remember using Solaris 10 back in 2005, wow it’s been a while!

They’re calling it the first cloud OS. I can’t say I really agree with that, vSphere, and even ESX before that has been more cloudy than Solaris for many years now, and remains today.

While their Xen-based Oracle VM is still included in Solaris 11, the focus clearly seems to be Solaris Zones, which, as far as I know is a more advanced version of User mode linux (which seems to be abandoned now?).

Zones, and UML are nothing new, Zones having been first released more than six years ago. It’s certainly a different approach to a full hypervisor approach so has less overhead, but overall I believe is an outdated approach to utility computing (using the term cloud computing makes me feel sick).

Oracle Solaris Zones virtualization scales up to hundreds of zones per physical node at a 15x lower overhead than VMware and without artificial limits on memory, network, CPU and storage resources.

It’s an interesting strategy, and a fairly unique one in today’s world, so it should give Oracle some differentiation.  I have been following the Xen bandwagon off and on for many years and never felt it a compelling platform, without a re-write. Red Hat, SuSE and several other open source folks have basically abandoned Xen at this point and now it seems Oracle is shifting focus away from Xen as well.

I don’t see many new organizations gravitating towards Solaris zones that aren’t Solaris users already (or at least have Solaris expertise in house), if they haven’t switched by now…

New, integrated network virtualization allows customers to create high-performance, low-cost data center topologies within a single OS instance for ultimate flexibility, bandwidth control and observability.

The terms ultimate flexibility and single OS instance seem to be in conflict here.

The efficiency of modern hypervisors is to the point now where the overhead doesn’t matter in probably 98% of cases. The other 2% can be handled by running jobs on physical hardware. I still don’t believe I would run a hypervisor on workloads that are truely hardware bound, ones that really exploit the performance of the underlying hardware. Those are few and far between outside of specialist niches these days though, I had one about a year and a half ago, but haven’t come across one since.

 

Travel, Data centers, cars

Filed under: Random Thought — Nate @ 5:03 am

It’s about 85 hours until I start my next road trip up to Seattle, I have been thinking about it and I think this is my most anticipated “vacation” I’ve gone on in as long as I can remember. I will be having a lot of fun during the 3 short days I will be in town for.

My next trip is going to be to Atlanta, which is in less than a month. I’m going to visit one of the largest data centers in the world, to install some equipment for my company. I visited a smaller data center outside of Seattle that weighed in at around 500,000 square feet, though I only saw a small portion of it, never really got to see the scale of the place because it was very closed off(the portion I was visiting was sub leased by Internap). Maybe they expanded it recently since they claim 1.2M square feet of data center space now….

So if you have any suggestions for places to eat or drink or hang out while I’m in Atlanta let me know, I’m not sure yet how much spare time I’ll have or how long I’ll be in town for. I have one friend in Atlanta that I plan to see while I am there, he said he thinks he has some equipment in the same facility.

Now a short update on my car. One of my friends said I should post an update on my car situation given that I bought a new car earlier this year, an uncommon one at that. It’s been almost 9 months and 10,000 miles.

10,000 miles on the odometer

The Good

It’s still very fun to drive, the torque vectoring all wheel drive corners like nobody’s business, really grips the road good. It’s very easy to park, being smaller than my previous SUV, and the rear parking camera helps a ton as well in that department. I routinely park in spots I would never even CONSIDER even TRYING to park in with my previous vehicle (which was a 2001 Nissan Pathfinder). I have had no issues with the car to-date.

The Bad

Nothing major. I suppose my biggest complaints for day-to-day use is the user interface to the stereo system. It’s only crashed once which is nice. My main complaints revolve around MP3 meta data. The stereo does not remember any of the meta data between restarts, it takes about 3-4 minutes to re-generate the meta data from the ID3 tags after a restart (it does pick up where it left off when it starts as far as music goes though — although song ordering is messed up at that point). The user interface also becomes completely unresponsive for a good four seconds when a new song loads so it can load the meta data for that song. I don’t know why this is, I mean my $40 portable MP3 players do a better job than the car stereo does at this point. The Garmin-powered navigation works quite well though. I also cannot access the video inputs when the car is not in park, which is annoying. I read on some sites it’s state law that you can’t do things like watch DVDs while driving so they have a connection to the parking brake that disables the video when the car is not in park. What I’d like to see more though is the rear camera on the road though, I think it would give a good viewing angle for changing lanes but I can’t get to that either.

Another minor complaint is my sub woofer in the trunk. It sounds awesome, it’s just a whole lot bigger than I was expecting when I asked Car Toys to install it. I really thought it was going to be flush with the floor allowing near full use of the trunk. But it is not flush it sticks up quite a bit, and thus I lose quite a bit of trunk capacity(the trunk as-is is already really small). For the most part it’s not a big deal though I rarely need the trunk. I have gotten used to using the back seats for my shopping.

I was really worried when I was moving to California, if I would have enough space for the last of my crap after the movers left(and my two cats). With the back seats down though I had more than enough space(was I relieved!).

The sound and navigation system(and backup camera) was all after market(which I paid a lot for), so it’s not really related to the car directly since it wasn’t included with the car.

Another minor complaint remains the lack of an arm wrest but I often rest my elbow on the passenger seat. Also the “tounge” of my sneakers have a habbit of triggering my gas cap to open when I leave the car on occasion.

The biggest drawback to buying a new car and moving to California less than a year after I bought it was .. buying a car and moving to California less than a year after I bought it.

I took my sweet time to register my car in this state, I didn’t check to see what the laws were but thought I had something like 60-90 days or whatever. In any case I didn’t register it until later, the last week of October to be precise. Technically I have 20 days to register the car I learned.

So what was my registration fees ? I asked my sister and she said “$49?” Not $49, not $99, not $299, not $599.

About $2,200. I think the actual registration was around $200. Then I had another $200 in late fees, THEN I had $1,500 in sales taxes (?!) Apparently the law in California now says that if you buy a new car and move here less than a year afterwards you have to pay CA state sales tax on the car (minus any taxes you paid in the origin state). A YEAR. The DMV person said it used to be 90 days, but they recently extended it to a year. That, of course I was not expecting at all. I showed them my original receipt and apparently I paid about $400 in WA state sales tax, and I owed CA nearly four times that. That certainly seems unfair, both to me and to WA. Not only that but they backdated my registration to August.

Apparently my CA drivers license which I had when I moved to WA in 2000 expired in 2003, so when I went to get a new drivers license they said it was technically a renewal instead of a new license so the test to take was shorter (which I barely passed, missing 3 out of a maximum of 3 questions, another person in front of me missed quite a bit more, she missed about 10 of 20 questions total).

I’ve kept my WA state plates on for now since I’m going back to WA this week for a few days.

I suppose the gas mileage isn’t as high as I was expecting it was going to be, with an official rating of 27 city, 32 highway I think I get closer to 20-21 city, and maybe 25 highway. It’s not a big deal though I didn’t buy this car thinking it would be a hybrid, it still gets quite a bit better milage than my Pathfinder which on an absolutely perfect day on the highway would get about 19.9, city I’m thinking more 10-12 range (premium gas for both). I push my Juke much harder than my Pathfinder, getting up to 3-4500 RPM on a regular basis, sometimes even 5,000+. I don’t think I ever pushed my Pathfinder beyond 4,000 RPM (didn’t try). I waste quite a bit of gas as the Juke encourages me to drive faster, that’s fine for me though I don’t mind.

The Funny/Strange

I’ve run into probably 15 other Jukes in my travels in the past 9 months. It’s strange because when I have someone in the car with me and I see a Juke I point it out, and get pretty excited. I see others do the same to me quite often as well (assuming they are in a Juke too).  I haven’t seen any in California for a month or so. One day I saw at least two the same day.

Lots o Miles

So even though I walk to work every day (I live 0.4 miles from the office), I still managed to put on quite a few miles so far, and about to put a whole lot more on. With about 2,000 miles to/from Seattle, and may be making a 2nd trip next week to Orange County which is about another 1,000 miles total.

So much for that ‘short update’ huh!

November 10, 2011

World’s fastest switch

Filed under: Networking — Tags: — Nate @ 8:42 pm

I came across this yesterday which is both a video, and more importantly an in-depth report on the about-to-be-released Black Diamond X-series switch. I have written a few times on this topic, I don’t have much that is really new, but then I ran across this PDF which has something I have been looking for – a better diagram on how this new next generation fabric is hooked up.

Up until now, most (all?) chassis switches relied on backplanes, or more modern systems used mid planes to transmit their electrical signals between the modules in the chassis.

Something I learned a couple of years ago (I’m not an electrical engineer) is that there are physical limits as to how fast you can push those electrons over those back and mid planes. There are serious distance limitations which makes the engineering ever more complicated the faster you push the system. Here I was thinking just crank up the clock speeds and make it go faster, but apparently it doesn’t work quite that way 🙂

For the longest time all of Extreme’s products were backplane based. Then they released a mid plane based product the Black Diamond 20808 a couple of years ago. This product was discontinued earlier this year when the X-series was announced. The 20808 had (in my simple mind) a similar design to what Force10 had been doing for many years – which is basically N+1 switch fabric modules (I believe the 20808 could go to something like 5 fabric modules), all of their previous switches had what they called MSMs, or Management Switch Modules. These were combination switch fabric and management modules., with a maximum of two per system, each providing half of the switch’s fabric capacity. Some other manufacturers like Cisco separated out their switch fabric from their management module. Having separate fabric modules really doesn’t buy you much when you only have two modules in the system. But if your architecture can go to many more (I seem to recall Force10 at one point having something like 8), then of course you can get faster performance. Another key point in the design is having separate slots for your switch fabric modules so they don’t consume space that would otherwise be used by ethernet ports.

Anyways, on the Black Diamond 20808 they did something else they had never done before, they put modules on both the front, AND on the back of the chassis. On top of that the modules were criss-crossed. The modules on the front were vertical, the modules on the back were horizontal. This is purely guessing here but I speculate the reason for that is, in part, to cut the distance needed to travel between the fabric and the switch ports. HP’s c-Class Blade enclosure has a similar mid plane design with criss crossed components. Speaking of which, I wonder if the next generation 3PAR will leverage the same “non stop” midplane technology of the c-Class. The 5 Terabits of capacity on the c-Class is almost an order of magnitude more than what is even available on the 3PAR V800. Whether or not the storage system needs that much fabric is another question.

Black Diamond 20808 (rear)

The 20808 product seemed to be geared more towards service providers and not towards high density enterprise or data center computing(if I remember right the most you could get out of the box was 64x10GbE ports which you can now get in a 1U X670V).

Black Diamond 20808 (front)

Their (now very old) Black Diamond 8000 series (with the 8900 model which came out a couple of years ago being the latest incarnation) has been the enterprise workhorse for them for many years, with a plethora of different modules and switch fabric options. The Black Diamond 8900 is a backplane based product.  I remember when it came out too – it was just a couple months after I bought my Black Diamond 10808s, in the middle of 2005. Although if I remember right the Black Diamond 8800, as it was originally released, did not support the virtual router capability that the 10808 supported that I intended to base my network design on. Nor did it support the Clear Flow security rules engine. Support for these features was added years later.

You can see the impact distance has on the Black Diamond 8900 for example, with the smaller 6-slot chassis getting at least 48Gbps more switching capacity per line card than the 10-slot chassis simply because it is smaller. Remember this is a backplane designed probably seven years ago, so it doesn’t have as much fabric capacity as a modern mid plane based system.

Anyways, back on topic, the Black Diamond X-series. Extreme’s engineers obviously saw the physics (?) limits they were likely going to hit when building a next generation platform and decided to re-think how the system works, resulting, in my opinion a pretty revolutionary way of building a switch fabric (at least I’m not aware of anything else like it myself). While much of the rest of the world is working with mid planes for their latest generation of systems, here we have the Direct Orthogonal Data Path Mating System or DOD PMS (yeah, right).

Black Diamond X-Series fabric

What got me started down this path, was I was on the Data Center Knowledge web site, and just happened to see a Juniper Qfabric advertisement. I’ve heard some interesting things about Qfabric since it was announced, it sounds similar to the Brocade VCS technology. I was browsing through some of their data sheets and white papers and it came across as something that’s really complicated. It’s meant to be simple, and it probably is, but the way they explain it to me at least makes it sound really complicated. Anyways I went to look at their big 40GbE switch which is at the core of their Qfabric interconnect technology. It certainly looks like a respectable switch from a performance stand point – 128 40GbE ports, 10 Terabits of switching fabric, weighs in at over 600 pounds(I think Juniper packs their chassis products with lead weights to make them feel more robust).

So back to the report that they posted. The networking industry doesn’t have anything like the SPC-1 or SpecSFS standardized benchmarks to measure performance, and most people would have a really hard time generating enough traffic to tax these high end switches.There is standard equipment that does it, but it’s very expensive.

So, to a certain extent you have to trust the manufacturer as to the specifications of the product, a way many manufacturers try to prove their claims of performance or latency is to hire “independent” testers to run tests on the products and give reports. This is one of those reports.

Reading it made me smile, seeing how well the X-Series performed but in the grand scheme of things it didn’t surprise me given the design of the system and the fabric capacity it has built into it.

The BDX8 breaks all of our previous records in core switch testing from performance, latency, power consumption, port density and packaging design. The BDX8 is based upon the latest Broadcom merchant silicon chipset.

For the Fall Lippis/Ixia test, we populated the Extreme Networks BlackDiamond® X8 with 256 10GbE ports and 24 40GbE ports, thirty three percent of its capacity. This was the highest capacity switch tested during the entire series of Lippis/Ixia cloud network test at iSimCity to date.

We tested and measured the BDX8 in both cut through and store and forward modes in an effort to understand the difference these latency measurements offer. Further, latest merchant silicon forward packets in store and forward for smaller packets, while larger packets are forwarded in cut-through making this new generation of switches hybrid cut-through/store and forward devices.

Reading through the latency numbers, they looked impressive, but I really had nothing to compare them with, so I don’t know how good. Surely for any network I’ll ever be on it’d be way more than enough.

The BDX8 forwards packets ten to six times faster than other core switches we’ve tested.

[..]

The Extreme Networks BDX8 did not use HOL blocking which means that as the 10GbE and 40GbE ports on the BDX8 became congested, it did not impact the performance of other ports. There was no back pressure detected. The BDX8 did send flow control frames to the Ixia test gear signaling it to slow down the rate of incoming traffic flow.

Back pressure? What an interesting term for a network device.

The BDX8 delivered the fastest IP Multicast performance measured to date being able to forward IP Multicast packets between 3 and 13 times faster then previous core switch measures of similar 10GbE density.

The Extreme Networks BDX8 performed very well under cloud simulation conditions by delivering 100% aggregated throughput while processing a large combination of east-west and north-south traffic flows. Zero packet loss was observed as its latency stayed under 4.6 μs and 4.8 μs measured in cut through and store and forward modes respectively. This measurement also breaks all previous records as the BDX8 is between 2 and 10 times faster in forwarding cloud based protocols under load.

[..]

While these are the lowest Watts/10GbE port and highest TEER values observed for core switches, the Extreme Networks BDX8’s actual Watts/10GbE port is actually lower; we estimate approximately 5 Watts/10GbE port when fully populated with 768 10GbE or 192 40GbE ports. During the Lippis/Ixia test, the BDX8 was only populated to a third of its port capacity but equipped with power supplies, fans, management and switch fabric modules for full port density population. Therefore, when this full power capacity is divided across a fully populated BDX8, its WattsATIS per 10GbE Port will be lower than the measurement observed [which was 8.1W/port]

They also mention the cost of power, and the % of list price that cost is, so we can do some extrapolation. I suspect the list price of the product is not final, and I am assuming the prices they are naming are based on the configuration they are testing with rather than a fully loaded system(which as mentioned above the switch was configured with enough fabric and power for the entire chassis but only ~50% of the port capacity was installed).

Anyways, they say the price to power it over 3 years is $10,424.05 and say that is less than 1.7% of it’s list price. Extrapolating that a bit I can guesstimate that the list price of this system as tested with 352 10GbE ports is roughly $612,000, or about $1,741 per 10GbE port.

The Broadcom technology is available to the competition, the real question is how long will it take for the competition develop something that can compete with this 20 Terabit switching fabric, which seems to be about twice as fast as anything else currently on the market.

HP has been working on some next generation stuff, I read about their optical switching technology earlier this year that their labs are working on, sounds pretty cool.

[..] Charles thinks this is likely to be sometime in the next 3-to-5 years.

So, nothing on the immediate horizon on that front.

November 8, 2011

EMC and their quad core processors

Filed under: Storage — Tags: , , , — Nate @ 8:48 am

I first heard that Fujitsu had storage maybe one and a half years ago, someone told me that Fujitsu was one company that was seriously interested in buying Exanet at the time, which caused me to go look at their storage, I had no idea they had storage systems. Even today I really never see anyone mention them anywhere, my 3PAR reps say they never encounter Fujitsu in the field(at least in these territories they suspect over in Europe they go head to head more often).

Anyways, EMC folks seem to be trying to attack the high end Fujitsu system, saying it’s not “enterprise”, in the end the main leg that EMC has trying to hold on to what in their eyes is “enterprise” is mainframe connectivity, which Fujitsu rightly tries to debunk that myth since there are a lot of organizations that are consider themselves “enterprise” that don’t have any mainframes. It’s just stupid, but EMC doesn’t really have any other excuses.

What prompted me to write this, more than anything else was this

One can scale from one to eight engines (or even beyond in a short timeframe), from 16 to 128 four-core CPUs, from two to 16 backend- and front-end directors, all with up to 16 ports.

The four core CPUs is what gets me. What a waste! I have no doubt that in EMC’s  (short time frame)  they will be migrating to quad socket 10 core CPUs right? After all, unlike someone like 3PAR who can benefit from a purpose built ASIC to accelerate their storage, EMC has to rely entirely on software. After seeing SPC-1 results for HDS’s VSP, I suspect the numbers for VMAX wouldn’t be much more impressive.

My main point is, and this just drives me mad. These big manufacturers touting the Intel CPU drum and then not exploiting the platform to it’s fullest extent. Quad core CPUs came out in 2007. When EMC released the VMAX in 2009, apparently Intel’s latest and greatest was still quad core. But here we are, practically 2012 and they’re still not onto at LEAST hex core yet? This is Intel architecture, it’s not that complicated. I’m not sure what quad core CPUs specifically are in the VMAX, but the upgrade from Xeon 5500 to Xeon 5600 for the most part was

  1. Flash bios (if needed to support new CPU)
  2. Turn box off
  3. Pull out old CPU(s)
  4. Put in new CPU(s)
  5. Turn box on
  6. Get back to work

That’s the point of using general purpose CPUs!! You don’t need to pour 3 years of R&D into something to upgrade the processor.

What I’d like to see, something I mentioned in a comment recently is a quad socket design for these storage systems. Modern CPUs have had integrated memory controllers for a long time now (well only available on Intel since the Xeon 5500). So as you add more processors you add more memory too. (Side note: the documentation for VMAX seems to imply a quad socket design for a VMAX engine but I suspect it is two dual socket systems since the Intel CPUs EMC is likely using are not quad-socket capable). This page claims the VMAX uses the ancient Intel 5400 processors, which if I remember right was the first generation quad cores I had in my HP DL380 G5s many eons ago. If true, it’s even more obsolete than I thought!

Why not 8 socket? or more? Well cost mainly. The R&D involved in an 8-socket design I believe is quite a bit higher, and the amount of physical space required is high as well. With quad socket blades common place, and even some vendors having quad socket 1U systems, the price point and physical size related to quad socket designs is well within reach of storage systems.

So the point is on these high end storage systems you start out with a single socket populated on a quad socket board with associated memory. Want to go faster? add another CPU and associated memory? Go faster still? add two more CPUs and associated memory (though I think it’s technically possible to run 3 CPUs, well there have been 3 CPU systems in the past, it seems common/standard to add them in pairs). Your spending probably at LEAST a quarter million for this system initially, probably more than that, the incremental cost of R&D to go quad socket given this is Intel after all is minimal.

Currently VMAX goes to 8 engines, they say they will expand that to more. 3PAR took the opposite approach, saying while their system is not as clustered as a VMAX is (not their words), they feel such a tightly integrated system (theirs included) becomes more vulnerable to “something bad happening” that impacts the system as a whole, more controllers is more complexity. Which makes some sense. EMC’s design is even more vulnerable being that it’s so tightly integrated with the shared memory and stuff.

3PAR V-Class Cluster Architecture with low cost high speed passive backplane with point to point connections totalling 96 Gigabytes/second of throughput

3PAR goes even further in their design to isolate things – like completely separating control cache which is used for the operating system that powers the controllers and for the control data on top of it, with the data cache, which as you can see in the diagram below is only connected to the ASICs, not to the Intel CPUs. On top of that they separate the control data flow from the regular data flow as well.

One reason I have never been a fan of “stacking” or “virtual chassis” on switches is the very same reason, I’d rather have independent components that are not tightly integrated in the event “something bad” takes down the entire “stack”. Now if your running with two independent stacks, so that one full stack can fail without an issue then that works around that issue, but most people don’t seem to do that. The chances of such a failure happening are low, but they are higher than something causing all of the switches to fail if the switches were not stacked.

One exception might be some problems related to STP which some people may feel they need when operating multiple switches. I’ll answer that by saying I haven’t used STP in more than 8 years, so there have been ways to build a network with lots of devices without using STP for a very long time now. The networking industry recently has made it sound like this is something new.

Same with storage.

So back to 3PAR. 3PAR changed their approach in their V-series of arrays, for the first time in the company’s history they decided to include TWO ASICs in each controller, effectively doubling the I/O processing abilities of the controller. Fewer, more powerful controllers. A 4-node V400 will likely outperform an 8-node T800. Given the system’s age, I suspect a 2-node V400 would probably be on par with an 8-node S800 (released around 2003 if I remember right).

3PAR V-Series ASIC/CPU/PCI/Memory Architecture

EMC is not alone, and not the worst abuser here though. I can cut them maybe a LITTLE slack given the VMAX was released in 2009. I can’t cut any slack to NetApp though. They recently released some new SPEC SFS results, which among other things, disclosed that their high end 6240 storage system is using quad core Intel E5540 processors. So basically a dual proc quad core system. And their lower end system is — wait for it — dual proc dual core.

Oh I can’t describe how frustrated that makes me, these companies touting using general purpose CPUs and then going out of their way to cripple their systems. It would cost NetApp all of maybe what $1200 to upgrade their low end box to quad cores? Maybe $2500 for both controllers? But no they rather you spend an extra, what $50,000-$100,000  to get that functionality?

I have to knock NetApp more to some extent since these storage systems are significantly newer than the VMAX, but I knock them less because they don’t champion the Intel CPUs as much as EMC does, that I have seen at least.

3PAR is not a golden child either, their latest V800 storage system uses — wait for it — quad core processors as well. Which is just as disgraceful. I can cut 3PAR more slack because their ASIC is what provides the horsepower on their boxes, not the Intel processors, but still that is no excuse for not using at LEAST 6 core processors. While I cannot determine precisely which Intel CPUs 3PAR is using, I know they are not using Intel CPUs because they are ultra low power since the clock speeds are 2.8Ghz.

Storage companies aren’t alone here, load balancing companies like F5 Networks and Citrix do the same thing. Citrix is better than F5 in their offering software “upgrades” on their platform that unlock additional throughput. Without the upgrade you have full reign of all of the CPU cores on the box which allow you to run more expensive software features that would normally otherwise impact CPU performance. To do this on F5 you have to buy the next bigger box.

Back to Fujitsu storage for a moment, their high end box certainly seems like a very respectable system with regards to paper numbers anyways. I found it very interesting the comment on the original article that mentioned Fujitsu can run the system’s maximum capacity behind a single pair of controllers if the customer wanted to, of course the controllers couldn’t drive all the I/O but it is nice to see the capacity not so tightly integrated to the controllers like it is on the VMAX or even on the 3PAR platform. Especially when it comes to SATA drives which aren’t known for high amounts of I/O, higher end storage systems such as the recently mentioned HDS, 3PAR and even VMAX tap out in “maximum capacity” long before they tap out in I/O if your loading the system with tons of SATA disks. It looks like Fujitsu can get up to 4.2PB of space leaving, again HDS, 3PAR and EMC in the dust. (Capacity utilization is another story of course).

With Fujitsu’s ability to scale the DX8700 to 8 controllers, 128 fibre channel interfaces, 2,700 drives and 512GB of cache that is quite a force to be reckoned with. No sub-disk distributed RAID, no ASIC acceleration, but I can certainly see how someone would be willing to put the DX8700 up against a VMAX.

EMC was way late to the 2+ controller hybrid modular/purpose built game and is still playing catch up. As I said to Dell last year, put your money where your mouth is and publish SPC-1 results for your VMAX, EMC.

With EMC so in love with Intel I have to wonder how hard they had to fight off Intel from encouraging EMC to use the Itanium processor in their arrays instead of Xeons. Or has Intel given up completely on Itanium now (which, again we have to thank AMD for – without AMD’s x86-64 extensions the Xeon processor line would of been dead and buried many years ago).

For insight to what a 128-CPU core Intel-based storage system may perform in SPC-1, you can look to this system from China.

(I added a couple diagrams, I don’t have enough graphics on this site)

November 4, 2011

Hitachi VSP SPC-1 Results posted

Filed under: Storage — Tags: , , , , — Nate @ 7:41 pm

[(More) Minor updates since original post] I don’t think I’ll ever work for a company that is in a position to need to leverage something like a VSP, but that doesn’t stop me from being interested as to how it performs. After all it has been almost four years since Hitachi released results for their USP-V, which had, at the time very impressive numbers(record breaking even, only to be dethroned by the 3PAR T800 in 2008) from a performance perspective(not so much from a cost perspective not surprisingly).

So naturally, ever since Hitachi released their VSP (which is OEM’d by HP as their P9500)about a year ago I have been very curious as to how well it performs, it certainly sounded like a very impressive system on paper with regards to performance. After all it can scale to 2,000 disks and a full terabyte of cache. I read an interesting white paper (I guess you could call it that) recently on the VSP out of curiosity.

One of the things that sort of stood out is the claim of being purpose built, they’re still using a lot of custom components and monolithic architecture on the system, vs most of the rest of the industry which is trying to be more of a hybrid of modular commodity and purpose built, EMC VMAX is a good example of this trend. The white paper, which was released about a year ago, even notes

There are many storage subsystems on the market but only few can be considered as real hi-end or tier-1. EMC announced its VMAX clustered subsystem based on clustered servers in April 2009, but it is often still offering its predecessor, the DMX, as the first choice. It is rare in the industry that a company does not start ramping the new product approximately half a year after launching. Why EMC is not pushing its newer product more than a year after its announcement remains a mystery. The VMAX scales up by adding disks (if there are empty slots in a module) or adding modules, the latter of which are significantly more expensive. EMC does not participate in SPC benchmarks.

The other interesting aspect of the VSP is it’s dual chassis design(each chassis in a separate rack), which, the white paper says is not a cluster, unlike a VMAX or a 3PAR system(3PAR isn’t even a cluster the way VMAX is a cluster intentionally with the belief the more isolated design would lead to higher availability). I would assume this is in response to earlier criticisms of the USP-V design in which the virtualization layer was not redundant(when I learned that I was honestly pretty shocked) – Hitachi later rectified the situation on the USP-V by adding some magic sauce that allowed you to link a pair of them together.

Anyways with all this fancy stuff, obviously I was pretty interested when I noticed they had released SPC-1 numbers for their VSP recently. Surely, customers don’t go out and buy a VSP because of the raw performance, but because they may need to leverage the virtualization layers to abstract other arrays, or perhaps the mainframe connectivity, or maybe they get a comfortable feeling knowing that Hitachi has a guarantee on the array where they will compensate you in the event there is data loss that is found to be the fault of the array (I believe the guarantee only covers data loss and not downtime, but am not completely sure). After all, it’s the only storage system on the planet that has such a stamp of approval from the manufacturer (earlier high end HDS systems had the same guarantee).

Whatever the reason,  performance is still a factor given the amount of cache and the sheer number of drives the system supports.

One thing that is probably never a factor is ease of use – since the USP/VSP are complex beasts to manage, something your very likely need significant amounts of training. One story I remember being told is a local HDS rep in the Seattle area mentioned to a 3PAR customer “after a few short weeks of training you’ll feel as comfortable on our platform as you do on 3PAR”. Something like that, the customer said “you just made my point for me”. Something like that anyways.

Would the VSP dethrone the HP P10000 ? I found the timing of the release of the numbers an interesting coincidence after all, I mean the VSP is over a year old at this point, why release now ?

So I opened the PDF, and hesitated .. what would be the results? I do love 3PAR stuff and I still view them as  somewhat of an underdog even though they are under the HP umbrella.

Wow, was I surprised at the results.

Not. Even. Close.

The VSP’s performance was not much better than that of the USP-V which was released more than four years ago. The performance itself is not bad but it really puts the 3PAR V800 performance in some perspective:

  • VSP -   ~270,000 IOPS @ $8.18/IOP
  • V800 – ~450,000 IOPS @ $6.59/IOP

But it doesn’t stop there

  • VSP -  ~$45,600 per usable TB (39% unused storage ratio)
  • V800 – ~$13,200 per usable TB (14% unused storage ratio)

Hitachi managed to squeeze in just below the limit for unused storage – which is not allowed to be above 45%, sounds kind of familiar. The VSP had only 17TB additional usable capacity than the original USP-V as tested. This really shows how revolutionary sub disk distributed RAID is for getting higher capacity utilization out of your system, and why I was quite disappointed when the VSP came out without anything resembling it.

Hitachi managed to improve their cost per IOP on the VSP vs the USP but their cost per usable TB has skyrocketed, about triple the price of the original USP-V results. I wonder why this is ? (I mis counted the digits in my original calculations, in fact the VSP is significantly cheaper than the USP when it was tested!) One big change in the VSP is the move to 2.5″ disk drives. The decision to use 2.5″ drives did kind of hamper results I believe since this is not a SPC-1/E Energy Efficiency test so power usage was never touted. But the largest 2.5″ 15k RPM drive that is available for this platform is 146GB(which is what was used).

One of the customers(I presume) at the HP Storage presentation I was at recently was kind of dogging 3PAR saying the VSP needs less power per rack than the T or V class(which requires 4x208V 30A per fully loaded rack).

I presume it needs less power because of the 2.5″ drives, also overall the drive chassis on 3PAR boxes do draw quite a bit of power by themselves(200W empty on T/V, 150W empty on F – given the T/V hold 250% more drives/chassis it’s a pretty nice upgrade).

Though to me especially on a system like this, power usage is the last thing I would care about. The savings from disk space would pay for the next 10 years of power on the V800 vs the VSP.

My last big 3PAR array cut the number of spindles in half and the power usage in half vs the array it replaced, while giving us roughly 50% better performance at the same time and the same amount of usable storage. So knowing that, and the efficiency in general I’m much less concerned as to how much power a rack requires.

So Hitachi can get 384 x 2.5″ 15k RPM disks in a single rack, and draw less power than 3PAR can get with 320 disks in a single rack.

You could also think of it this way: Hitachi can get ~56TB RAW of 15k RPM space in a single rack, and 3PAR can get 192TB RAW of 15k RPM space in a single rack, nearly four times the RAW space for double the power(nearly five times the usable(4.72x precisely – V800 @ 80% utilization VSP @ 58%) space due to the architecture), for me that’s a good trade off, a no brainer really.

The things I am not clear on with regards to these results – is this the best the VSP can do?  This does appear to be a dual chassis system. The VSP supports 2,000 spindles, the system tested only had 1,152, which is not much more than the 1,024 tested on the original USP-V. Also the VSP supports up to 1TB of cache however the system tested only had 512GB (3PAR V800 had 512GB of data cache too).

Maybe it is one of those situations where this is the most optimal bang for the buck on the platform,  perhaps the controllers just don’t have enough horsepower to push 2,000 spindles at full tilt – I don’t know.

Hitachi may have purpose built hardware, but it doesn’t seem to do them a whole lot of good when it comes to raw performance. I don’t know about you but I’d honestly feel let down if I was an HDS customer. Where is the incentive to upgrade from USP-V  to VSP ? The cost for usable storage is far higher, the performance is not much better. Cost per IOP is less, but I suspect the USP-V at this stage of the game with more current pricing for disks and stuff would be even more competitive than VSP. (correction from above, due to my mis-reading of the digits ($135k/TB instead of $13.5k/TB VSP is in fact cheaper making it a good upgrade from USP from that perspective! Sorry for the error 🙂 )

Maybe it’s in the software – I’ve of course never used a USP/VSP system but perhaps the VSP has newer software that is somehow not backwards compatible with the USP, though I would not expect this to be the case.

Complexity is still high, the configuration of the array stretches from page 67 of the full disclosure documentation to page 100. 3PAR’s configuration by contrast is roughly a single page.

Why do you think someone would want to upgrade from a USP-V to a VSP?

I still look forward to the day when the likes of EMC make their VMAX architecture across their entire product line, and HDS also unifies their architecture. I don’t know if it’ll ever happen but it should be possible at least with VMAX given their sound thumping of the Intel CPU drum set.

The HDS AMS 2000 series of products was by no means compelling either, when you could get higher availability(by means of persistent cache with 4 controllers), three times the amount of usable storage, and about the same performance on a 3PAR F400 for about the same amount of money.

Come on EMC, show us some VMAX SPC-1 love, you are, after all a member of SPC now (though kind of odd your logo doesn’t show up, just the text of the company name).

One thing that has me wondering on this VSP configuration – with such little disk space available on the system I have to wonder why anyone would bother with such a configuration with spinning rust on this platform for performance and just go SSD instead. Well one reason may be a 400GB SSD would run $35-40,000 (after 50% discount, assuming that is list price), ouch. 220 of those (system maximum is 256) would net you roughly 88TB raw (maybe 40TB usable), but cost $7.7M for the SSDs alone.

On the V800 side a 4-pack of 200GB SSDs will cost you roughly $25,000 after discount(assuming that is list price). Fully loading a V800 with the maximum of 384 SSDs(77TB raw) would cost roughly $2.4M for the SSDs alone(and still consume 7 racks) I think I like the 3PAR SSD strategy more  — not that I would ever do such a configuration!

Space would be more of course if the SSD tests used RAID 5 instead of RAID 1, I just used RAID 1 for simplicity.

Goes to show that these big boxes have a long way to go before they can truly leverage the raw performance of SSD. with 200-300 SSDs in either of these boxes the controllers would be maxed out and the SSDs would probably for the most part be idle. I’ve been saying for a while how stupid it seems for such a big storage system to be overwhelmed so completely by SSDs and that the storage systems need to be an order of magnitude(or more) faster. 3PAR did a good start with doubling of performance, but I’m thinking the boxes need to do at least 1M IOPS to disk, if not 2M, for systems such as these — how long will it take to get there?

Maybe HDS could show better results with 900GB 10k RPM disks(and put something like 1700 disks instead of 1,152 instead of the 146GB 15k RPM disks, should hopefully show much lower per usable TB costs, though their IOPS cost would probably shoot up. Though from what I can see, 600GB is the max supported 2.5″ 10k RPM disk supported. 900GB @ 58% utilization would yield about 522GB, and 600GB on 3PAR @ 80% utilization would yield about 480GB.

I suspect they could get their costs down significantly more if they went old school and supported larger 3.5″ 15k RPM disks and used those instead(the VSP supports 3.5″ disks just nothing other than 2TB Nearline (7200 RPM) and 400GB Flash – a curious decision). If you were a customer isn’t that something you’d be interested in ? Though another compromise you make with 3.5″ disks on the VSP is your then limited to 1,280 spindles, rather than 2,048 with 2.5″. Though this could be a side effect of a maximum addressable capacity of 2.5PB which is reached with 1,280 2TB disks, they very well could probably support 2,048 3.5″ disks if they could fit in the 2.5PB limit.

Their documentation says actual maximum usable with Nearline 2TB disks is 1.256PB (with RAID 10). With 58% capacity utilization that 1.25PB drops dramatically to 742TB. With RAID 5 (7+1) they say 2.19PB usable, let’s take that 58% number again 1.2PB @ 58% capacity utilization.

V800 by contrast would top out at 640TB with RAID 10 @ 80% utilization (stupid raw capacity limits!! *shakes fist at sky*),  or somewhere in the 880TB-1.04PB (@80%) range with RAID 5 (depending on data/parity ratio).

Another strange decision on both HDS and 3PAR’s part was to only certify the 2TB disk on their platform, and not anything smaller. Since both systems become tapped out at roughly half the number of supported spindles when using 2TB disks due to capacity limits.

An even more creative solution(to work around the limitation!!), which I doubt is possible is somehow restrict the addressable capacity of each disk to 1/2 the size, so you could in effect get 1,600 2TB disks in each with 1TB of capacity, then when they release software updates to scale the box to even higher levels of capacity (32GB of control cache can EASILY handle more capacity) just unlock the drives and get that extra capacity free. That would be nice at least, probably won’t happen though. I bet it’s technically possible on the 3PAR platform due to their chunklets. 3PAR in general doesn’t seem as interested in creative solutions as I am 🙂 (I’m sure HDS is even more rigid)

Bottom Line

So HDS has managed to get their cost for usable space down quite a bit, from $135,000/TB to around $45,000/TB, and improve performance by about 25%

They still have a long ways to go in the areas of efficiency, performance, scalability, simplicity and unified architecture. I’m sure there will be plenty of customers out there that don’t care about those things(or are just not completely informed) and will continue to buy VSP for other reasons, it’s still a solid platform if your willing to make those trade offs.

Mass defections away from Vmware coming?

Filed under: Virtualization — Tags: — Nate @ 10:57 am

I have expected as much since Vmware announced their abrupt licensing changes, in the same survey that I commented on last night for another reason, another site has reported on another aspect of it – nearly 40% of respondents are strongly considering moving away from Vmware in the coming year, 47% of which cite the licensing charges as the cause.

A Gartner analyst questions the numbers saying the move will be more complicated than people think and that will help Vmware retain share. I don’t agree with that myself I suspect for most customers the move will probably not be complex at all.

Myself I was just recently trying to a dig a bit more into KVM trying to figure out what they use for storage, it seems for block based systems they are using GFS2 (can’t find the link off hand)?  Though I imagine they can run on top of NFS too. I wonder what the typical deployment is for KVM when it comes to storage – is shared storage widely used or is it instead used mostly with local DAS?

I just read an interesting comment from a Xen user (I’ve never found Xen to be a compelling platform myself from a technology perspective, my own personal use of Xen has been mostly indirect by means of EC2 – which in general is an absolutely terrible experience), from a thread on slashdot about this topic –

Hyper-V is about 5 years behind and XenServer is about 3 years behind in terms of functionality and stability, mainly due to the fact that VMWare has been doing it for so long. VMWare is rock-solid and feature rich, and I’d love to use them. Currently we use XenServer, but with Citrix recently closing down their hardware API’s and not playing nicely with anyone it looks like it is going to be the first casualty. I’ve been very upset by XenServer’s HA so far, plain and simple it has sucked. I’ve had hosts reboot from crashes and the virtual machines go down, but the host thinks it has the machines and all of the other hosts think it has the machines. I’ve done everything XenServer has asked (HA quorum on a separate LUN, patches, etc), but it still just sucks. I’ve yet to see a host fail and the machines to go elsewhere, and the configuration is absolutely right and has been reviewed by Citrix. Maybe 6.0 will be better, but I just heard of major issues today with it. Hyper-V is really where the competition is going to come from, especially with how engrained it is in everything coming up. Want to run Exchange 2010 SP2? Recommendation is Hyper-V virtual machines.

God I miss VMWare.

I hope Vmware comes through for me and produces a price point for the basic vSphere services that is more cost effective(basically I’d like to see vSphere Standard edition with say something crazy like 256GB/socket vRAM with the current pricing). Though I’d settle for with whatever vRAM is available in enterprise plus.

So your actually paying more for the features.

I can certainly find ways to “make do” at a cost of $1,318/socket (w/1 year of enterprise support based on this pricing), for Standard edition (includes Vmotion and HA), vs $4,369/socket for Enterprise plus. Two sockets would be around $2,600 — which is less than where vSphere 3 was, which was in the $3,000-3,500 range per pair of sockets for standard edition in 2007.

I’m not holding my breath though(since being kicked in the teeth with vSphere 5 licensing changes).

Time will tell if there are such defections, unlike Netflix where the commitment is basically zero, we’ll have to wait for the next round of hardware refreshes to kick in to see what sort of impact there is from the licensing change. Speaking of hardware refreshes(that need vSphere 5) what the hell is taking so long with the Opteron 6200s, AMD?! I really thought they’d show up in September, then couldn’t imagine them not showing up in October, and here we are at November, and still no word.

Vmware does need a “Netflix moment”, a term that has been used quite a bit recently.

« Newer Posts — Older Posts »

Powered by WordPress