TechOpsGuys.com Diggin' technology every day

August 23, 2011

Mac Daddy P10000

Filed under: Datacenter,Storage,Virtualization — Tags: , , — Nate @ 9:55 pm

It’s finally here, the HP P10000 – aka 3PAR V Class. 3PAR first revealed this to their customers more than a year ago, but the eagle has landed now.

When it comes to the hardware – bigger is better (usually means faster too)

Comparisons of recent 3PAR arrays

ArrayRaw
Capacity
Fibre
Ports
Data
Cache
Control
Cache
DisksInterconnect
Bandwidth
I/O
Bandwidth
SPC-1
IOPS
8-node P10000
(aka V800)
1,600 TB288 ports
(192 host)
512 GB256 GB1,920112 GB/sec96 GB/sec600,000
(guess)
8-node T800800 TB192 ports
(128 host)
96 GB32 GB1,28045 GB/sec19.2 GB/sec225,000
4-node T800
(or 4-node
T400)
400 TB96
(64 host)
48 GB 16 GB6409.6 GB/sec?~112,000
(estimate)
4-node F400384 TB32
(24 host)
24 GB16 GB3849.6 GB/sec ??93,000
Comparison between the F400, T400, T800 and the new V800. In all cases the numbers reflected are in a maximum configuration.

3PAR V800 ready to fight

The new system is based on their latest Generation 4 ASIC, and for the first time they are putting two ASICs in each controller. This is also the first system that supports PCI Express, with if my memory serves 9 PCI Express buses per controller. Front end throughput is expected to be up in the 15 Gigabytes/second range (up from ~6GB on the T800).  Just think they have nearly eight times the interconnect bandwidth than the controllers have capacity to push data to hosts, that’s just insane.

IOPS – HP apparently is not in a big rush to post SPC-1 numbers, but given the increased spindle count, cache, doubling up on ASICs, and the new ASIC design itself I would be surprised if the system would get less than say half a million IOPS on SPC-1 (by no means a perfect benchmark but at least it’s a level playing field).

It’s nice to see 3PAR finally bulk up on data cache (beefcake!!) – I mean traditionally they don’t need it all that much because their architecture blows the competition out of the water without breaking a sweat – but still – ram is cheap – it’s not as if they’re using the same type of memory you find in CPU cache – it’s industry standard ECC DIMMs. RAM may be cheap, but I’m sure HP won’t charge you industry standard DIMM pricing when you go to put 512GB in your system!

Now that they have PCI Express 3PAR can natively support 8Gbps fibre channel as well as 10Gbit iSCSI and FCoE which are coming soon.

The drive cages and magazines are more or less unchanged (physically) from the previous generation but apparently new stuff is still coming down the pike there.  The controller’s physical design (how it fits in the cabinet) seems radically different than their previous S or T series.

Another enhancement for this system is they expanded the number of drive chassis to 48, or 12 per node (up from 8 per node). Though if you go back in time you’ll find their earliest S800 actually supported 64 drive chassis for a time, since then they have refrained from daisy chaining drive chassis on their S/T/V class which is how they achieved the original 64 drive chassis configuration (or 2,560 disks back when disks were 9GB in size). The V class obviously has more ports so they can support more cages. I have no doubt they could go to even more cages by using ports assigned to hosts and assign them to disks, just a matter of testing. Flipping a fiber port from host to disk is pretty trivial on the system.

The raw capacity doesn’t quite line up with the massive amount of control cache the system has, in theory at least if 4GB of control cache per controller is good enough for 200TB raw (per controller pair), then 32GB  per controller should be able to net you 1,600 TB raw (per controller pair or 6,400 TB for the whole system), but obviously with a limit put in of 1,600 TB for the entire system they are using a lot of control cache for something else.

As far as I know the T-class isn’t going anywhere anytime soon, this V class is all about even more massive scale, at a significantly higher entry level price point than the T-class(at least $100,000 more at the baseline from what I can tell), with the beauty of running the same operating system, the same user interfaces, the same software features across the entire product line. The T-class, as-is still is mind numbingly fast and efficient, even three years after it was released.

No mainframe connectivity on this baby.

Storage Federation

The storage federation stuff is pretty cool in that it is peer based, you don’t need any external appliances to move the data around, the arrays talk to each other directly to manage all of that. This is where we get the first real integration between 3PAR and HP in that the entire line of 3PAR arrays as well as the Lefthand-based P4000 iSCSI systems (including the Virtual storage appliance even!) support this new peer federation (sort of makes me wonder where EVA support is – perhaps it’s coming later or maybe it’s a sign HP is sort of depreciating EVA when it comes to this sort of thing – I’m sure the official party line will be EVA is still a shining star).

The main advantage I think of storage federation technology over something like storage vMotion is the array has a more holistic view of what’s going on in the storage system rather than just what a particular host sees, or what a particular LUN is doing. The federation should also have more information about the location of the various arrays if they are in another data center or something and make more intelligent choices about moving stuff around. Certainly would like to see it in action myself. Even though hypervisors have had thin provisioning for a while – by no means does it reduce the need for thin provisioning at the storage level (at least for larger deployments).

I’d imagine like most things on the platform the storage federation is licensed based on the capacity of the array.

If this sort of thing interests you anywhere nearly as much as it interests me you should check out the architecture white paper from HP which has some new stuff from the V class here. You don’t have to register to download it like you did back in the good ‘ol days.

I’d be surprised if I ever decided to work for a company large enough to be able to leverage a V-class, but if anyone from 3PAR is out there reading this (I’m sure there’s more than one) since I am in the Bay area – not far from your HQ – I wouldn’t turn down an invitation to see one of these in person 🙂

Oh HP.. first you kick me in the teeth by killing WebOS devices then before I know what happened you come out with a V-class and want to make things all better, I just don’t know what to feel.

The joys of working with a 3PAR array, it’s been about a year since I laid my hands on one (working at a different company now), I do miss it.

June 29, 2011

Oracle picks up Pillar

Filed under: Storage — Tags: , — Nate @ 1:56 pm

Most people have been expecting this for a long time, and have wondered why it didn’t happen sooner, with Oracle ditching HDS as an OEM partner almost immediately after acquiring Sun.

I have read, and heard over the past year that Oracle has been for the most part destroyed in the storage market (servers doing badly as well) as a result since their Sun storage products just are not competitive. Many larger customers have been leaving to the likes of HP and IBM who could offer the “one stop shop” for servers and storage (even before HP bought 3PAR, HP had and still has their OEM’d HDS equipment).

In some informal talks with some HDS folks last year they seemed quite happy that Oracle was no longer an OEM, saying that the people over at Sun/Oracle weren’t competent enough to handle the HDS stuff (*cough* too complicated *cough*), and so HDS just went in direct with most of those customers that Oracle walked away from.

Finally someone at Oracle woke up and realized there still is, and will continue to be for some time a big market for traditional SAN systems, far bigger than the market of customers willing to risk putting their data on cheap SATA controllers on servers running ZFS with high failure rates and poor performance.

So it finally happened, Oracle is buying Pillar. At first look however it really does seem like an odd scenario, from their SEC filing

The Earn-Out therefore will only be paid to Mr. Ellison, his affiliates and, if applicable, to the other Pillar Data stockholders and option holders if the Net Revenues during Year 3 of the Earn-Out Period exceed the Net Losses, if any, during the entire Earn-Out Period.

There’s no specific mention whether or not Larry is going to pay himself back for the $500M+ in loans he has given to Pillar over the years, so I suppose not. In any case it won’t be until the end of 2014 when we might discover what value Oracle has placed on Pillar. One commenter on The Register mentions Pillar’s revenue as $29M per year, don’t know where that came from though, doing some searching myself I found references to roughly $70M in revenue, to $3B in revenue (if that was the case they would of IPO’d)

I think it’s a good deal for Pillar to, they get much better validation on their products in front of customers.

I’ve gone through quite a bit of the information on the Pillar web site and to-date I have not seen anything that would make me want to buy their product, and have yet to hear any positive words coming from the people I know in the street/industry (granted my community is limited).

But it sure as hell beats anything that Oracle has been offering their customers recently, that alone may be enough to drive a decent amount of sales.

Pillar posted some updated SPC-1 numbers recently, a significant improvement over their original numbers, though nothing ground breaking from a competitive standpoint.

In other news, two early social media giants have fallen – MySpace being acquired for $35M, and Friendster re-inventing itself as a gaming site with Facebook authentication. I’d bet the infrastructure behind Myspace is worth about $35M by itself – Newscorp really wanted out!

April 26, 2011

Misleading 3PAR

Filed under: Storage — Tags: , — Nate @ 8:43 am

Hello to my two readers out there!

You know I like 3PAR, have been using them for years, and know their stuff inside and out.

I was on a Computerworld article a few moments ago and saw an advertisement for 3PAR by HP and it made me cringe

While it is true that 3PAR has Intel Xeon processors, it’s really the custom built ASIC that does all the heavy lifting. General purpose CPUs don’t have a prayer in being able to keep up, much like general purpose CPUs don’t have a prayer in keeping up with high performance network switching fabrics.

I think the advertisement is bad, and misleading (by misleading I mean removing perceived value of the 3PAR platform by implying that Intel processors are the workhorse on the system). I’m sure that 3PAR would of never had made this mistake on their own. Someone at HP needs to be educated on the platform.

So I have to knock HP on that one.

March 9, 2011

Next Gen COPAN

Filed under: Storage — Tags: , — Nate @ 9:41 am

About a year or so ago SGI bought COPAN for what seemed like fractional pennies on the dollar, well they recently came out with the next generation of COPAN and I’m still amazed at how much storage they can fit in a rack.

ArcFiniti comes in 5 factory-configured models to suit any archive environment. Lower-capacity models can be upgraded to higher capacity, maxing out at just over 1.4PB of usable archive in a single rack.

Full specifications don’t seem to be disclosed at the moment, the original COPAN systems topped out at a hefty 3,000 pounds per rack, the only storage system that I had heard of that weighed in more than 3PAR (about 2,000 pounds max per rack).

The original systems kept roughly 75% of the drives spun down at any given point.

 

March 2, 2011

Compellent gets Hyper efficient storage tiering

Filed under: Storage — Tags: , , , , — Nate @ 9:24 am

So according to this article from our friends at The Register, Compellent is considering going to absurdly efficient storage tiering taking the size of data being migrated to 32kB from their currently insanely efficient 512kB.

That’s just insane!

For reference, as far as I know:

  • 3PAR moves data around in 128MB chunks
  • IBM moves data around in 1GB chunks (someone mentioned that XIV uses 1MB)
  • EMC moves data around in 1GB chunks
  • Hitachi moves data around in 42MB chunks (I believe this is the same data size they use for allocating storage to thin provisioned volumes)
  • NetApp has no automagic storage tiering functionality though they do have PAM cards which they claim is better.

I have to admit I do like Compellent’s approach the best here, hopefully 3PAR can follow. I know 3PAR allocates data to think provisioned volumes in 16kB chunks, what I don’t know is whether or not their system is adjustable to get down to a more granular level of storage tiering.

There’s just no excuse for the inefficient IBM and EMC systems though, really, none.

Time will tell if Compellent actually follows through with going as granular as 32kB, I can’t help but suspect the CPU overhead of monitoring so many things will be too much for the system to bear.

Maybe if they had purpose built ASIC…

 

February 19, 2011

Flash not good for offline storage?

Filed under: Random Thought,Storage — Tags: , , — Nate @ 9:36 am

A few days ago I came across an article on Datacenter Knowledge that was talking about Flash reliability. As much as I’d love to think that just because it’s solid state that it will last much longer, real world tests to-date haven’t shown that to be true in many cases.

I happened to have the manual open on my computer for the Seagate Pulsar SSD, and just saw something that was really interesting to me, on page 15 it says –

As NAND Flash devices age with use, the capability of the media to retain a programmed value begins to deteriorate. This deterioration is affected by the number of times a particular memory cell is programmed and subsequently erased. When a device is new, it has a powered off data retention capability of up to ten years. With use the retention capability of the device is reduced. Temperature also has an effect on how long a Flash component can retain its pro-grammed value with power removed. At high temperature the retention capabilities of the device are reduced. Data retention is not an issue with power applied to the SSD. The SSD drive contains firmware and hardware features that can monitor and refresh memory cells when power is applied.

I am of course not an expert in this kind of stuff, so was operating under the assumption that if the data is written then it’s written and won’t get  “lost” if it is turned off for an extended period of time.

Seagate rates their Pulsar to retain data for up to one year without power at a temperature of 25 C (77 F).

Compare to what tape can do. 15-30 years of data retention.

Not that I think that SSD is a cost effective method to do backups!

I don’t know what other manufacturers can do, I’m not picking on Seagate, but found the data tidbit really interesting.

(I originally had the manual open to try to find reliability/warranty specs on the drive to illustrate that many SSDs are not expected to last multiple decades as the original article suggested).

December 12, 2010

Dell and Exanet: MIA

Filed under: Storage — Tags: , — Nate @ 9:37 pm

The thoughts around Dell buying Compellent made me think back to Dell’s acquistiion of the IP and some engineering employees of Exanet, as The Register put it, a crashed NAS company.

I was a customer and user of Exanet gear for more than a year, and at least in my experience it was a solid product, very easy to use, decent performance and scalable. The back end architecture to some extent mirrored the 3PAR hardware-based architecture but in software, really a good design in my opinion.

Basic Exanet Architecture

Their standard server at the time they went under was a IBM x3650, dual proc quad core Intel Xeon 5500-based platform with 24GB of memory.

Each server had multiple software processes called fsds or File system daemons, that ran, they ran one fsd per core. Each fsd was responsible for a portion of the file system (x number of files), they load balanced it quite well I never had to manually re-balance or anything. Each fsd was allocated its own memory space used for itself as well as cache, if I recall right the default was around 1.6GB per fsd.

Each NAS head unit had back end connectivity to all of the other NAS units in the cluster(minimum 2, maximum tested at the time they went under was 16). A request for a file could come in on any node, any link. If the file wasn’t home to that node it would transparently forward the request to the right node/fsd to service the request on the back end. Much like how 3PAR’s backplane forwards requests between controllers.

Standard for back end network was 10Gbps on their last models.

As far as data protection, the use of “commodity” servers did have one downside, they had to use UPS systems as their battery backup to ensure enough time for the nodes to shut down cleanly in the event of a power failure. This could present problems at some data centers as operating a UPS in your own rack can be complicated from a co-location point of view(think EPO etc). Another similar design that Exanet had compared to 3PAR is their use of internal disks to flush cache to, which is something I suppose Exanet was forced into doing, other storage manufacturers use battery backed cache in order to survive power outages of some duration. But both Exanet and 3PAR dump their cache to an internal disk so that the power outage can last for a day, a week, or even a month and it won’t matter, data itnegrity is not compromised.

32-bit platform

The only thing that held it back was they didn’t have enough time or resources to make the system fully 64-bit before they went under, that would of unlocked a whole lot of additional performance they could of gotten. Being locked into a 32-bit OS really limited what they could do on a single node, and as processors became ever more powerful they really had to make the jump to 64-bit.

Exanet was entirely based on “commodity” hardware, not only were they using x86 CPUs but their NAS controllers were IBM 2U rackmount servers running CentOS 4.4 or 4.5 if I recall right.

To me, as previous posts have implied, if your going to base your stuff on x86 CPUs, go all out, it’s cheap anyways. I would of loved to have seen a 32-48 core Exanet NAS controller with 512GB-1TB of memory on it.

Back to Dell

Dell originally went into talks with Exanet a while back because Exanet was willing to certify Equallogic storage as a back end provider of disk to an Exanet cluster, using iSCSI inbetween the Exanet cluster and the Equallogic storage. Since nobody else in the indusry seemed willing to have their NAS solution talk to a back end iSCSI system. As far as I know the basic qualifications for this solution was completed in 2009, quite a ways before they ran out of cash.

Why did Exanet go under? I believe primarily because the market they were playing in was too small with too few players in it, not enough deals to go around, so whomever had the most resources to outlast the rest would come out on top, in this case I believe it was Isilon, even though they too were taken out by EMC from the looks of their growth it didn’t seem like they were in a fine position to continue to operate independently. With Ibrix and Polyserve going to HP, Onstor going to LSI, and I’m still convinced BlueArc will go to HDS at some point(they are once again filing for IPO but word on the street is they aren’t in very good shape), I suspect after they fail to IPO and go under. They have a very nice NAS platform, but HDS has their hands tied in supporting 3rd party storage other than HDS product, BlueArc OEM’s LSI storage like so many others.

About a year ago SGI OEM’d one of BlueArc’s products though recently I have looked around the SGI site and see no mention of it. Either they have abandoned it (more likely) or are just really quiet. Since I know SGI is also a big LSI shop I wonder if they are making the switch to Onstor. One industry insider I know suspects LSI is working on integrating the Onstor technology directly into their storage systems rather than having an independent head unit, which makes sense if they can make it work.

But really my question is why hasn’t Dell announced anything related to the Exanet technology? They could of, quite possibly within a week or two had a system running and certified on Dell PowerEdge equipment and selling to both existing Exanet customers as well as new ones. The technology worked fine, it was really easy to setup and use, and it’s not as if Dell has another solution in house that competes with it. AND since it was an entirely software based solution there was really no costs involved in manufacturing. Exanet had more than one PB-sized deal in the works at the time they went under, that’s a lot of good will Dell just threw away. But hey, what do you expect, it’s Dell. Thankfully they didn’t get their dirty paws on 3PAR.

When I looked at how a NetApp system was managed compared to the Exanet my only response was You’re kidding, right?

Time will tell if anything ever comes of the technology.

I really wanted 3PAR to buy them of course, they were very close partners with 3PAR and both pitched each other’s products at every opportunitiy. Exanet would go out of their way to push 3PAR storage whenever possible because they knew how much trouble the LSI storage could be, and they were happy to get double the performance per spindle off 3PAR vs LSI. But I never did get an adequate answer out of 3PAR as to why they did not pursue Exanet, they were in the early running but pulled out for whatever reason, the price tag of less then $15M was a steal.

Now that 3PAR is with HP we’ll see what they can do with Ibrix, I knew of more than one customer that migrated off of things like Ibrix and Onstor to Exanet, HP has been pretty silent about Ibrix since they bought them as far as I know. I have no idea how much R&D they have pumped into it over the years or what their plans might be.

Dell going after Compellent

Filed under: Storage — Tags: , — Nate @ 12:26 am

I know this first made news a couple of days ago but I can’t tell you how busy I’ve been recently. It seems like after Dell got reamed by HP in the 3PAR bidding war they are going after Compellent,  one of the only other storage technology companies utilizing distributed RAID, and as far as I know the main pioneer of automagic storage tiering.

This time around nobody else is expected to bid, it seems the stock speculators were a bit disappointed when the talks were announced as they had already bid the stock up far higher than what is being discussed as being the acquisition price.

While their previous generation of controllers seemed rather weak, their latest and greatest look to be a pretty sizable step up, and apparently can be leveraged by their existing customers, no need to buy a new storage system.

I can’t wait to see how EMC responds myself. Dell must be really frustrated with them to go after Compellent so soon after losing 3PAR.

November 24, 2010

More inefficient storage

Filed under: Storage — Tags: , , — Nate @ 8:32 am

Another random thought, got woken up this morning and wound up checking what’s new on SPC-1, and a couple weeks ago the Chinese company Huawei posted results for their Oceanspace 8100 8-node storage system. This system seems to be similar to the likes of HDS USP/VSP, IBM SVC in that it has the ability to virtualize other storage systems behind it. The system is powered by 32 quad core processors or 128 CPU cores.

The thing that caught my eye is in every SPC-1 disclosure is the paragraph

Unused Storage Ratio: Total Unused Capacity (XXX GB) divided by Physical
Storage Capacity (XXX GB) and may not exceed 45%.

So what is Huawei’s Unused storage ratio? – 44.77%

I wonder how hard it was for them to get under the 45% limit, I bet they were probably at 55-60% and had to yank a bunch of drives out or something to decrease their ratio.

From their full disclosure document it appears their tested system has roughly 261TB of unused storage on it. That’s pretty bad, 3PAR F400 has a mere 75GB of unused capacity (0.14%) by contrast. The bigger T800 has roughly 21TB of unused capacity (15%).

One would think, that for Huawei, they would be better off using 146GB disks instead of the 300GB, 450GB and 600GB disks (another question is what is the point in mismatched disks for this test, maybe they didn’t have enough of one drive type which would be odd for a drive array manufacturer – maybe they mixed drive types to drive the unused capacity perhaps after having started with nothing but 600GB disks).

Speaking of drive sizes, one company I know well has a lot of big Oracle databases and are I/O bound more than space bound, so it benefits them to use smaller disk drives, their current array manufacturer no longer offers 146GB disk drives so they are forced to pay quite a bit more for the bigger disks.

Lots of IOPS to be sure, 300,000 of them (260 IOPS per drive) and 320GB of cache (see note below!), but certainly seems that you could do this a better way..

Looking deeper into the full disclosure documents(Appendix C page 64) for the Huawei system reveals this little gem

The creatlun command creates a LUN with a capacity 1,716,606 MiB. The -p 0 parameter, in the creatlun command sets the read cache policy as no prefetch and the -m 0 parameter sets the write cache policy as write cache with no mirroring.

So they seem to be effectively disabling the read cache and disabling cache mirroring making all cache a write back cache that is not protected? I would imagine they ran the test and found their read cache ineffective so disabled it and devoted it to write cache and re-ran the test.

Submitting results without mirrored cache seems, well misleading to say the least. Glad there is full disclosure!

The approximate cost of the Huawei system seems to be about $2.2 million according to the google exchange rate.

While I am here, what is it with 8 node storage systems? What is magical about that number? I’ve seen a bunch of different ones both SAN and NAS that top out at eight. Not 10? not 6? Seems a strange coincidence, and has always bugged me for some reason.

November 15, 2010

ISilon gets taken out by EMC

Filed under: News,Storage — Tags: — Nate @ 9:02 am

Looks like EMC did it after all, buying Isilon for $2.25 Billion. Probably the biggest tech deal for the Seattle area for quite some time.

I haven’t paid too much attention to Isilon recently but it does seem like they have a nice product for the scale out media space, lots of big files and high throughput. Isilon, along with Panasas seem to be unique in tightly integrating the storage controller with the NAS side, while other solutions were purely gateway approaches of one sort or another.

So who’s next?

 

« Newer PostsOlder Posts »

Powered by WordPress