TechOpsGuys.com Diggin' technology every day

29Jun/11Off

Oracle picks up Pillar

TechOps Guy: Nate

Most people have been expecting this for a long time, and have wondered why it didn't happen sooner, with Oracle ditching HDS as an OEM partner almost immediately after acquiring Sun.

I have read, and heard over the past year that Oracle has been for the most part destroyed in the storage market (servers doing badly as well) as a result since their Sun storage products just are not competitive. Many larger customers have been leaving to the likes of HP and IBM who could offer the "one stop shop" for servers and storage (even before HP bought 3PAR, HP had and still has their OEM'd HDS equipment).

In some informal talks with some HDS folks last year they seemed quite happy that Oracle was no longer an OEM, saying that the people over at Sun/Oracle weren't competent enough to handle the HDS stuff (*cough* too complicated *cough*), and so HDS just went in direct with most of those customers that Oracle walked away from.

Finally someone at Oracle woke up and realized there still is, and will continue to be for some time a big market for traditional SAN systems, far bigger than the market of customers willing to risk putting their data on cheap SATA controllers on servers running ZFS with high failure rates and poor performance.

So it finally happened, Oracle is buying Pillar. At first look however it really does seem like an odd scenario, from their SEC filing -

The Earn-Out therefore will only be paid to Mr. Ellison, his affiliates and, if applicable, to the other Pillar Data stockholders and option holders if the Net Revenues during Year 3 of the Earn-Out Period exceed the Net Losses, if any, during the entire Earn-Out Period.

There's no specific mention whether or not Larry is going to pay himself back for the $500M+ in loans he has given to Pillar over the years, so I suppose not. In any case it won't be until the end of 2014 when we might discover what value Oracle has placed on Pillar. One commenter on The Register mentions Pillar's revenue as $29M per year, don't know where that came from though, doing some searching myself I found references to roughly $70M in revenue, to $3B in revenue (if that was the case they would of IPO'd)

I think it's a good deal for Pillar to, they get much better validation on their products in front of customers.

I've gone through quite a bit of the information on the Pillar web site and to-date I have not seen anything that would make me want to buy their product, and have yet to hear any positive words coming from the people I know in the street/industry (granted my community is limited).

But it sure as hell beats anything that Oracle has been offering their customers recently, that alone may be enough to drive a decent amount of sales.

Pillar posted some updated SPC-1 numbers recently, a significant improvement over their original numbers, though nothing ground breaking from a competitive standpoint.

In other news, two early social media giants have fallen - MySpace being acquired for $35M, and Friendster re-inventing itself as a gaming site with Facebook authentication. I'd bet the infrastructure behind Myspace is worth about $35M by itself - Newscorp really wanted out!

Tagged as: , Comments Off
10Oct/10Off

Intel or ASIC

TechOps Guy: Nate

Just another one of my random thoughts I have been having recently.

Chuck wrote a blog not too long ago how he believes everyone is going to go to Intel (or x86 at least) processors in their systems and move away from ASICs.

He illustrated his point by saying some recent SPEC NFS results showed the Intel based system outperforming everything else. The results were impressive, the only flaw in them is that the costs are not disclosed for SPEC. An EMC VMAX with 96 EFDs isn't cheap. And the better your disk subsystem is the faster your front end can be.

Back when Exanet was still around they showed me some results from one of their customers testing SPEC SFS on the Exanet LSI (IBM OEM'd) back end storage vs 3PAR storage, and for the same number of disks the SPEC SFS results were twice as high on 3PAR.

But that's not my point here or question. A couple of years ago NetApp posted some pretty dismal results for the CX-3 with snapshots enabled. EMC doesn't do SPC-1 so NetApp did it for them. Interesting.

After writing up that Pillar article where I illustrated the massive efficiency gains on the 3PAR architecture(which is in part driven by their own custom ASICs), it got me thinking again, because as far as I can tell Pillar uses x86 CPUs.

Pillar offers multiple series of storage controllers to best meet the needs of your business and applications. The Axiom 600 Series 1 has dual-core processors and supports up to 24GB cache. The Axiom 600 Slammer Series 2 has quad-core processors and double the cache providing an increase in IOPS and throughput over the Slammer Series 1.

Now I can only assume they are using x86 processors, for all I know I suppose they could be using Power, or SPARC, but I doubt they are using ARM 🙂

Anyways back to the 3PAR architecture and their micro RAID design. I have written in the past about how you can have tens to hundreds of thousands of mini RAID arrays on a 3PAR system depending on the amount of space that you have. This is, of course to maximize distribution of data over all resources to maximize performance and predictability. When running RAID 5 or RAID 6, there are of course parity calculations involved. I can't help but wonder what sort of chances in hell a bunch of x86 CPU cores have in calculating RAID in real time for 100,000+ RAID arrays, with 3 and 4TB drives not too far out, you can take that 100,000+ and make it 500,000.

Taking the 3PAR F400 SPC-1 results as an example, here is my estimate on the number of RAID arrays on the system, fortunately it's mirrored so math is easier:

  • Usable capacity = 27,053 GB (27,702,272 MB)
  • Chunklet size = 256MB
  • Total Number of RAID-1 arrays = ~ 108,212
  • Total Number of data chunklets = ~216,424
  • Number of data chunklets per disk = ~563
  • Total data size per disk = ~144,128 MB (140.75 GB)

For legacy RAID designs it's probably not a big deal, but as disk drives grow ever bigger I have no doubt that everyone will have to move to a distributed RAID architecture, to reduce disk rebuild times and lower the chances of a double/triple disk failure wiping out your data. It is unfortunate (for them) that Hitachi could not pull that off in their latest system.

3PAR does use Intel CPUs in their systems as well, though they aren't used too heavily, on the systems I have had even at peak spindle load I never really saw CPU usage above 10%.

I think ASICs are here to stay for some time, on the low end you will be able to get by with generic CPU stuff, but on the higher end it will be worth the investment to do it in silicon.

Another place to look for generic CPUs vs ASICs is in the networking space. Network devices are still heavily dominated by ASICs because generic CPUs just can't keep up. Now of course generic CPUs are used for what I guess could be called "control data", but the heavy lifting is done by silicon. ASICs often draw a fraction of the power that generic CPUs do.

Yet another place to look for generic CPUs vs ASICs is in the HPC space - the rise of GPU-assisted HPC allowing them to scale to what was (to me anyways) unimaginable heights.

Generic CPUs are of course great to have and they have come a long way, but there is a lot of cruft in them, so it all depends on what your trying to accomplish.

The fastest NAS in the world is still BlueArc, which is powered by FPGAs, though their early cost structures put them out of reach for most folks, their new mid range looks nice, my only long time complaint about them has been their back end storage - either LSI or HDS, take it or leave it. So I leave it.

The only SPEC SFS results posted by BlueArc are for the mid range, nothing for their high end (which they tested on the last version of SFS, nothing yet for the current version).

 

9Oct/10Off

Capacity Utilization: Storage

TechOps Guy: Nate

So I was browsing through that little drop down address bar in Firefox hitting the sites I usually hit, and I decided hey let's go look at what Pillar is doing. I've never used their stuff but I dig technology you know, so I like to try to keep tabs on companies and products that I haven't used, and may never consider using, good to see what the competition is up to, because you never know they may come out with something good.

Tired of the thin argument

So the CEO of Pillar has a blog, and he went on a mini rant about how 3PAR^H^H^H^HHP is going around telling people you can get 100TB of capacity in one of their 70TB arrays. I haven't read too deep into what the actual claim they are making is, but being so absolutely well versed in 3P..HP technology I can comment with confidence in what their strategy is and how they can achieve those results. Whether or not they are effective at communicating that is another story, I don't know because well I don't read everything they say.

Pillar notes that HP is saying that due to the 3PAR technologies you can get by with less and he's tired of hearing that old story over and over.

Forget about thin for the moment!

So let me spray paint another angle for everyone to see. As you know I do follow SPC-1 numbers pretty carefully. Again not that I really use them to make decisions, I just find the numbers and disclosure behind them very interesting and entertaining at times. It is "fun" to see what others can do with their stuff in a way that can be compared on a level playing field.

I wrote, what I consider a good article on SPC-1 benchmarks a while back, EMC gave me some flak because they don't believe SPC-1 is a valid test, when I believe EMC just doesn't like the disclosure requirements, but I'm sure you won't ever hear EMC say that.

SPC-1 Results

So let's take the one and only number that Pillar published, because, well that's all I have to go on, I have no personal experience with their stuff, and don't know anyone that uses it. So if this information is wrong it's wrong because the results they submitted were wrong.

So, the Pillar Axiom 600's results have not stood the test of time well at all, as you would of noticed in my original article, but to highlight:

  • System tested: January 12, 2009
  • SPC-1 IOPS performance: 64,992
  • SPC-1 Usable space: 10TB
  • Disks used: 288 x 146G 15k RAID 1
  • IOPS per disk: 226 IOPS/disk
  • Average Latency at peak load: 20.92ms
  • Capacity Utilization (my own metric I just thought of): 34.72 GB/disk
  • Cost per usable TB (my own metric extrapolated from SPC-1): $57,097 per TB
  • Cost per IOP (SPC-1 publishes this): $8.79

The 3PAR F400 by contrast was tested just 105 days later and absolutely destroyed the Pillar numbers, and unlike the Pillar numbers the F400 has held up very well against the test of time all the way to present day even:

  • System tested: April 27, 2009
  • SPC-1 IOPS performance: 93,050
  • SPC-1 Usable space: 27 TB
  • Disks used: 384 x 146G 15k RAID 1
  • IOPS per disk: 242 IOPS/disk
  • Average Latency at peak load: 8.85ms
  • Capacity Utilization: 70.432 GB/disk
  • Cost per usable TB: $20,312 per TB
  • Cost per IOP: $5.89

Controller Capacity

Now in my original post I indicated stark differences in some configurations that tested substantially less physical disks than the controllers supported, there are a couple of possibilities I can think of for this:

  • The people running the test didn't have enough disks to test (less likely)
  • The controllers on the system couldn't scale beyond the configuration tested, so to illustrate the best bang for your $ they tested with the optimal number of spindles to maximize performance (more likely)

So in Pillar's case I think the latter is the case as they tested with a pretty small fraction of what their system is advertised as being capable of supporting.

Efficiency

So taking that into account, the 3PAR gives you 27TB of usable capacity, note here we aren't even taking into account the thin technologies, just throw those out the window for a moment, let's simplify this.

The Pillar system gives you 10TB of usable capacity, the 3PAR system gives you about 270% more space and 130% more performance for less money.

What would a Pillar system look like(or Systems I guess I should say since we need more than one) that could give us 27TB usable capacity and 93,000 SPC-1 IOPS using 146G 15k RPM disks (again trying to keep level playing field here)?

Well I can only really guess, to reach the same level of performance Pillar would need an extra 124 disks, so 412 spindles. Maintaining the same level of short stroking that they are doing(34.7GB/disk), those extra 124 spindles only get you to roughly 14.3TB.

And I'm assuming here because my comments earlier about optimal number of disks to achieve performance, if you wanted to get those extra 124 spindles in you need a 2nd Axiom 600, and all the costs with the extra controllers and stuff. Controllers obviously carry a hefty premium over the disk drives. While the costs are published in Pillar's results I don't want to spend the time to try to extrapolate that angle.

And if you do in fact need more controllers, the system was tested with two controllers, if you have to go to four (tested 3PAR F400 has four), 3PAR has another advantage completely unrelated to SPC-1, the ability to maintain performance levels under degraded conditions (controller failure, software upgrade, whatever) with Persistent Cache. Run your same SPC-1 test, and yank a controller out from each system (3PAR and Pillar) and see what the results are. The numbers would be even more embarrassingly in 3PAR's favor thanks to their architecture and this key caching feature. Unlike most of 3PAR's feature add-ons, this one comes at no cost to the customer, the only requirement is you must have at least 4 controllers on the box.

So you still need to get to 27 TB of usable capacity. From here it can get really fuzzy because  you need to add enough spindles to get that high but then you need to adjust the level of short stroking your doing to use more of the space per drive, it wouldn't surprise me if this wasn't even possible on the Pillar system(not sure if any system can do it really,  but I don't know).

If Pillar can't adjust the size of the short stroking then the numbers are easy, at 34.7GB/disk they need 778 drives to get to 27TB of usable capacity, roughly double what 3PAR has.

Of course the performance of a two-system based Axiom 600 with 778 drives will likely outperform a 384-disk F400(I should hope so at least), but you see where I'm going.

I'm sure Pillar engineers could come up with a way to configure the system more optimally my 778 drive solution is crazy but from a math perspective it's the easiest and quickest thing I could come up with, with the data I have available to me.

This is also a good illustration why when I go looking at what Xiotech posts, I really can't compare them against 3PAR or anybody else, because they only submit results for ~16 drive systems. To me, it is not valid to compare a ~16 drive system to something that has hundreds of drives and try to extrapolate results. Xiotech really does give stellar results as far as capacity utilization and IOPS/disk and stuff, but they haven't yet demonstrated that those numbers are scalable beyond a single ISE enclosure - yet alone to several hundred disks.

I also believe the 3PAR T800 results could be better too, the person at 3PAR who was responsible for running the test was new to the company at the time and the way he laid out the system was, odd to say the least. The commands he used were even depreciated. But 3PAR isn't all that interested in re-testing, they're still the record holder for spinning rust in a single system(more than two years running now no doubt!).

Better response times to boot

You can see the 3PAR system performs with less than half the amount of latency that the Pillar system does despite the Pillar system short stroking their disks. Distributed RAID with full mesh architecture at work baby. I didn't even mention it but the Pillar system has double the cache than than the F400. I mean the comparison really almost isn't fair.

I'm sure Pillar has bigger and better things out now since they released the SPC-1 numbers for the Axiom, so this post has the obvious caveat that I am reporting based on what is published. They'd need to pull more than a rabbit out of a hat to make up these massive gaps though I think.

Another Angle

We could look at this another way as well, assuming for simplicity's sake for a moment that both systems scale lineally up or down, we can configure a 3PAR F400 with the same performance specs as the Pillar that was tested.

You'd need 268 disks on the 3PAR F400 to match the performance of the Pillar system. With those 268 disks you'd get 18.4 TB of usable space, same performance, fewer disks, and 184% additional usable capacity. And scaling the cost down like we scaled the performance down, the cost would drop to roughly $374,000, a full $200,000 less than Pillar for the same performance and more space.

Conclusion

So hopefully this answers the question with more clarity why you can get less storage from the 3PAR F400 and get the same or better performance and usable capacity than going with a Pillar Axiom 600.  At the end of the day 3PAR drives higher capacity utilization and delivers superior results for significantly less greenbacks. And I didn't even take 3PAR's thin technologies into account, the math there can become even more fuzzy depending on the customer's actual requirements and how well they can leverage thin built in.

You may be able to understand why HP was willing to go to the end of the earth to acquire 3PAR technology. And you may be able to understand why I am so drawn to that very same technology. And here I'm just talking about performance. Something that unlike other things(ease of use etc) is really easy to put hard numbers on.

The numbers are pretty simple to understand, and you can see why the big cheese at HP responsible for spear heading the 3PAR purchase said:

The thin provisioning, automatic storage tiering, multi-tenancy, shared-memory architecture, and built-in workload management and load balancing in the 3PAR arrays are years ahead of the competition, according to Donatelli, and therefore justify the $2.4bn the company paid to acquire 3PAR in a bidding war with rival Dell.

Maybe if I'm lucky I can trigger interest in The Register again by starting a blog war or something and make tech news! woohoo! that would be cool. Of course now that I said that it probably won't happen.

I'm told by people who know the Pillar CEO he is "raw", much like me. so it will be interesting to see the response. I think the best thing they can do is post new SPC-1 numbers with whatever the latest technology they have is, preferably on 146G 15k disks!

Side note

It was in fact my 3PAR rep that inspired me to write about this SPC-1 stuff, I was having a conversation with him earlier in the year where he didn't think the F400 was as competitive against the HDS AMS2500 as he felt it needed to be. I pointed out to him that despite the AMS2500 having similar SPC-1 IOPS and similar cost, the F400 offered almost twice the usable capacity. And the cost per usable TB was far higher on the 2500. He didn't realize this.  I did see this angle so felt the need to illustrate it. Hence my Cost per SPC-1 Usable TB. It's not a performance metric, but in my opinion from a cost perspective a very essential metric, at least for highly efficient systems.

(In case it wasn't obvious, I am by no means compensated by 3PAR in any way for anything I write, I have a deep passion for technology and they have some really amazing technology, and they make it easy to use and cost effective to boot)