Diggin' technology every day

June 11, 2013

Pedal to the metal: HP 3PAR 7450

Filed under: Storage — Tags: — Nate @ 8:43 am

[NOTE: I expect to revise this many times – I’m not at HP Discover (maybe next year!), so I am basing this post off what info I have seen elsewhere, I haven’t yet got clarification on what NDA info specifically I can talk about yet so am trying to be cautious !]

[Update: HP’s website now has the info]

I was hoping they would announce the SPC-1 results of this new system, and I was going to wait until that happens, but I am not sure if they have them finalized yet, I’ve heard the ballpark figures, but am waiting for the official results.

The upside is I am on the east coast so I am up bright and early relative to my normal Pacific time zone morning.

I thought it would be announced later in the week but my first hint was this Russian blog (google translated), which I saw on LinkedIn a few minutes ago(relative to the time I started the blog post which took me a good two hours to write), also came across this press release of sorts, and there is the data sheet for the new system.

In a nut shell the 7450 is the system that HP mentioned at the launch event for the 7400 last December – though the model number was not revealed they said

In addition to mixed SSD/HDD and all-SSD configurations across the HP 3PAR StoreServ family, HP has announced the intent to develop an SSD-optimized hardware model based on the 3PAR operating system.

As fast as the all-SSD 7400 was, that was not the “optimized” hardware model – this one is (the one that was mentioned last December). I think the distinction with the word optimized vs using the phrase purpose built is important to keep in mind.

The changes from a hardware perspective are not revolutionary, 3PAR has, for the first time in their history (as far as I know anyway) has fairly quickly leveraged the x86 processors and upgraded both the processors and the memory (ASIC is the same as 7400) to provide the faster data ingest rate. I had previously (incorrectly of course) assumed that the ASIC was tapped out with earlier results and perhaps they would need even more ASICs to drive the I/O needs of an all-SSD system. The ASIC will be a bottleneck at some point but it doesn’t seem to be today – the bottleneck was the x86 CPUs.

They also beefed up the cache, doubling what the 7400 has.

  • 4-Node 7400: 4 x Intel Xeon 6-core 1.8 Ghz w/64GB Cache
  • 4-Node 7450: 4 x Intel Xeon 8-core 2.3Ghz w/128GB Cache

Would of been nice to have seen them use the 10-core chips, maybe the turnaround for such a change would of been too difficult to pull off in a short time frame. 8-core Intel is not bad though.

The Russian blog above touts a 55% increase in performance on the 7450 over the 7400, and the cost is about 6% more (the press release above quotes $99,000 as entry level pricing)

Throughput is touted as 5.5 Gigabytes/second, which won’t win any SPC-2 trophies, but is no slouch either – 3PAR has always been more about random IOPS than sequential throughput (though they often tout they can do both simultaneously within a single array – more effectively than other platforms).

The new system is currently tested (according to press release) at 540,000 read IOPS @ 0.6ms of latency. Obviously SPC-1 will be less than the 100% random read. This compares to the 7400 which was tested(under the same 100% read test I believe) to run at 320,000 IOPS @ 1.6ms of latency. So a 59.2% improvement in read IOPS and about 62% less latency.

Maybe we could extrapolate that number a bit here, the 7400 achieved 258,000 SPC-1 IOPS. 59.2% more would make the 7450 look like it would score around 413,000 SPC-1 IOPS, which is nearly the score of an 8-node P10800 which has 16 ASICs and 16 x Quad core Xeon processors! (that P10800 requires basically a full rack for just the controllers vs 4U for the 7450 (assuming they can get the full performance out of the controllers with only 48 SSD drives).

The blog also talks about the caching improvements targeted to improve performance and lifetime of the SSDs. The new 3PAR software also has a media wear gauge for the SSDs, something I believe the HP Lefthand stuff got in a year or two ago (better late than never!). The graphics the Russian blog has are quite good, I didn’t want to too shamelessly rip them from their blog to re-post here so I encourage you to go there to see the details on the caching improvements that are specific to SSD).

The competition

This system is meant to go head to head with the all-flash offerings from the likes of EMC, IBM NetApp (not aware of any optimized flash systems from HDS yet – maybe they will buy one of those new startups to fill that niche – they do have an optimized flash module for their VSP but I’d consider that a different class of product which may retain the IOPS constraints of the VSP platform).

However unlike the competition who has had to go outside of their core technology, HP 3PAR has been able to bring this all flash offering under the same architecture as the spinning rust models, basically it’s the same system with some tweaked software and faster processors with more memory. The underlying OS is the same, the features are the same, the administrative experience is the same. It’s the same, which is important to keep in mind. This is both good and bad, though for the moment I believe more good (Granted of course HP had to go to 3PAR to get all of this stuff, but as this blog has had a lot of 3PAR specific things I view this more in a 3PAR light than in a HP light if you get what I mean).

Of the four major competitors, EMC is the only one that touts deduplication (which, IMO is only really useful for things like VDI in transactional workloads)

3PAR is the only one with a mature enterprise/service provider grade operating system. On top of that obviously 3PAR is the only one that has a common platform amongst all of it’s systems from the 7200 all the way to the 10800.

3PAR and IBM are the only ones that are shipping now. Just confirmed from El Reg that the 7450 is available immediately.

None of big four tout compression, which I think would be a greater value add than deduplication for most workloads. I’m sure it’s on all of their minds though, it could be a non trivial performance hit, and in 3PAR’s case they’d likely need to implement it in the ASIC, if so, it means having to wait until the next iteration of the ASIC comes out. There has been gzip compression available in hardware form for many years so I imagine it wouldn’t be difficult to put into the silicon to keep the performance up under such conditions.

The new system also supports a 400GB MLC self encrypting drive (along with other SEDs for other 3PAR platforms as well) – 3PAR finally has a native encryption option, for those that need it.

Who should buy this

This isn’t an array for everyone (nor are the ones from the other big storage players). It’s a specialized system for specific very high performance workloads where latency is critical, yet at the same time providing the availability and manageability of the 3PAR platform to an all SSD solution.

You can probably go buy a server and stuff it with a few PCIe flash boards and meet or exceed the IOPS at a similar latency and maybe less price. If your workload is just dumb IOPS and you care about the most performance at the least price then there are other options available to you (they probably won’t work as well but you get what you (don’t) pay for).

There clearly is a market for such a product though, the first hint of this was dropped when HP announced an all flash version of it’s P10000 about a year ago. Customers really wanted an all flash system and they really wanted the 3PAR OS on it. If your not familiar with the high end 3PAR systems well from a form factor perspective driving 400k+ SPC-1 IOPS on a P10800 vs a 7450 you would probably get a good chuckle out of how much floor space and power circuits are required for the P10800 (power draw would be light on the SSDs of course, but they have hard requirements for power provisioning – most customers would pay per circuit regardless of draw).

I think a lot of this may be in the banking sector, where folks are happy to buy tons of fancy low latency stuff to make sure their transactions are processed in milliseconds.

Fifteen milliseconds may not seem like a significant amount of time—it is literally shorter than a human blink of an eye, which takes 300 to 400 milliseconds. But in the age of super-high-speed computerized trading, Wall Street firms need less than a millisecond to execute a trade.


All told, Nanex calculated that $28 million worth of shares were exchanged in a short time[15 milliseconds] before the official release of the ISM data.

The skeptics

There have been a lot of skeptics out there wondering whether or not the 3PAR architecture could be extended to cover an all flash offering (you can actually sort of count me in the skeptical camp as well, I was not sure even after they tried to re-assure me, I want to see the numbers at the end of the day). I believe with this announcement they have shown that even more so than the 7400, they have a very solid all flash offering that will, in most cases beat the tar out of the competition, not only on performance, not only on latency, not only on enterprise grade availability and functionality, but on price as well.

Even with this high performance system, these all SSD systems illustrate quite well how a modern storage controller is not able to scale anywhere nearly as well with SSDs as with spinning rust. Most of the SSD offerings have a small number of SSDs before they tap out the controllers. No single controller(that I’ve seen) supports the multi millions of IOPS that would be required to drive many hundreds of SSDs at line rate simultaneously(like regular storage arrays would drive hundreds of disks today).

It is just interesting to me to see the massive bottleneck shift continues to be the controller, and will be for some time to come. I wonder when the processors will get fast enough that they might shift the bottleneck back to the storage media, a decade? Or perhaps by that time everyone will be running on some sort of mature grid storage technology, and the notion of controllers as most of us know them today will be obsolete as a concept. Certainly several cloud providers are already trying to provide grid storage as an alternative, though in most cases, while the cost can be low, the performance is very poor as well (relative to an HP 3PAR anyway).

There is always more work to do (in this case mainly dedupe and compression), and as you might expect HP, along with the other big storage companies are constantly working to add more, I am very excited about what the future holds for 3PAR, really have never been so excited since the launch of the 7000 series last year(as a customer now for almost seven years) and am very pleased with what HP has managed to accomplish with the technology thus far.

Other 3PAR announcements

  • 3PAR Priority Optimization is made available now (first announced last December) – this is basically fine grained QoS for IOPS and throughput, something that will be a welcome enhancement to those running true multi tenant systems.
  • 3PAR Recovery Manager for Hyper-V – sounds like they are bringing Hyper-V up to the same level of support as VMware.
  • As mentioned earlier, Self encrypting drive options are cited on the Russian blog include – 400GB MLC SSD, 450GB 10k, 900GB 10k, 1TB 7.2k 2.5 “

Side note: there are a few other things to write about later, such as the IBM XIV SPC-1, the HP StoreOnce VSA, and probably whatever else comes out at Discover. For sure I won’t get to those today(or maybe even tomorrow, I am on a semi vacation/working week this week).

May 28, 2013

3PAR up 82% YoY – $1 Billion run rate

Filed under: Storage — Tags: — Nate @ 8:49 am

I came across this article on The Register which covered some of HP’s storage woes (short story: legacy storage is nose diving and 3PAR is shining). El Reg linked to the conference call transcript and I just ran a quick keyword search for 3PAR and saw this

This has been one of our most successful product introductions and 3PAR has now exceeded the $1 billion run-rate revenue mark.


Converged storage products were up 48% year-over-year and within that 3PAR was up 82%

Congratulations 3PAR! Woohoo! All of us over here at Techopsguys are really proud of you – keep up the good work! <voice=”Scotty”>Almost brings a tear to me eye.</voice>

For a comparison, I dug up the 3PAR results on for the quarter immediately previous to them being acquired:

3PAR® (NYSE: PAR), the leading global provider of utility storage, today reported results for the first quarter of fiscal year 2011, which ended June 30th, 2010. Revenue for the first quarter was $54.3 million, an increase of 22% as compared to revenue of $44.5 million for the same period in the prior year, and an increase of 1% as compared to $53.7 million in the prior quarter, which ended March 31st, 2010.

I can’t help but wonder how well Compellent is doing for Dell these days by contrast, since Dell withdrew from the bidding war for 3PAR with HP and went for them instead.. (side note: I once saw some value in Compellent as an alternative to 3PAR but that all went away with the 3PAR 7000-series.). I looked at the transcript for Dell’s latest conference call and the only thing they touch about storage is declines of 10%, no mention of any product lines as far as I could tell.

May 23, 2013

3PAR 7400 SSD SPC-1

Filed under: Storage — Tags: , , — Nate @ 10:41 am

I’ve been waiting to see these final results for a while, and now they are out! The numbers(performance + cost + latency) are actually better than I was expecting.

You can see a massive write up I did on this platform when it was released last year.

(last minute edits to add a new Huawei results that was released yesterday)

(more last minute edits to add a HP P6500 EVA SPC-1E)

SPC-1 Recap

I’ll say this again in case this happens to be read by someone who is new here. Myself, I see value in the SPC-1 as it provides a common playing field for reporting on performance in random transactional workloads (the vast majority of workloads are transactional). On top of the level playing field the more interesting stuff comes in the disclosures of the various vendors. You get to see things like

  • Cost (SpecSFS for example doesn’t provide this and the resulting claims from the vendors showing high performance relative to others at a massive cost premium but not disclosing the costs is very sad)
  • Utilization (SPC-1 minimum protected utilization is 55%)
  • Configuration complexity (only available in the longer full disclosure report)
  • Other compromises the vendor might of made (see the note about disabling cache mirroring)
  • 3 year 24×7 4 hour on site hardware support costs

There is a brief executive summary as well as what is normally a 50-75 page full disclosure report with the nitty gritty details.

SPC-1 also has maximum latency requirements – no I/O request can take longer than 30ms to serve or the test is invalid.

There is another test suite -  SPC-2, which tests throughput with various means. Much fewer systems participate in that test (3PAR never has, though I’d certainly like them to).

Having gone through several storage purchases over the years I can say from personal experience it is a huge pain to try to evaluate stuff under real workloads – often times vendors don’t even want to give evaluation gear (that is in fact in large part why I am a 3PAR customer today). Even if you do manage to get something in house to test, there are many things out there, with wide ranging performance / utilization ratios. At least with something like SPC-1 you can get some idea how the system performs relative to other systems at non trivial utilization rates. This example is rather extreme but is a good illustration.

I have no doubt the test is far from perfect, but in my opinion at least it’s far better than the alternatives, like people running 100% read tests with IOMeter to show they can get 1 million IOPS.

I find it quite strange that none of the new SSD startups have participated in SPC-1, I’ve talked to a couple different ones and they don’t like the test, they give the usual it’s not real world, customers should take the gear and test it out themselves. Typical stuff. Usually means they would score poorly – especially those that leverage SSD as a cache tier, with high utilization rates of SPC-1 you are quite likely to blow out that tier, once that happens performance tanks. I have heard reports of some of these guys getting their systems yanked out of production because they fail to perform after utilization goes up. System shines like a star during brief evaluation – then after several months of usage and utilization increasing, performance no longer holds up.

One person said their system is optimized for multiple workloads and SPC-1 is a single workload. I don’t really agree with that, SPC-1 does a ton of reads and writes all over the system, usually from multiple servers simultaneously. I look back to 3PAR specifically, who have been touting multiple workload (and mixed workload) support since their first array was released more than a decade ago. They have participated in SPC-1 for over a decade as well, so arguments saying testing is too expensive etc doesn’t hold water either. They did it when they were small, on systems that are designed from the ground up for multiple workloads (not just riding a wave of fast underlying storage and hoping that can carry them),  these new small folks can do it too. If they can come up with a better test with similar disclosures I’m all ears too.

3PAR Architecture with mixed workloads

The one place where I think SPC-1 could be improved is in failure testing. Testing a system in a degraded state to see how it performs.

The below results are from what I could find on all SSD SPC-1 results. If there is one/more I have missed(other than TMS, see note below), let me know. I did not include the IBM servers with SSD, since those are..servers.

Test Dates

HP 3PAR 7400May 23, 2013
HP P6500 EVA (SPC-1E)February 17, 2012
IBM Storwize V7000June 4, 2012
HDS Unified Storage 150March 26, 2013
Huawei OceanStor Dorado2100 G2May 22, 2013
Huawei OceanStor Dorado5100August 13, 2012

I left out the really old TMS (now IBM) SPC-1 results as they were from 2011, too old for a worthwhile comparison.

Performance / Latency

System NameSPC-1
Avg Latency
(all utilization
# of times
above 1ms
# of
HP 3PAR 7400258,0780.66ms0.86ms0 / 1532x
HP P6500 EVA (SPC-1E)20,0034.01ms11.23ms13 / 158x
IBM Storwize V7000120,4922.6ms4.32ms15 / 1518x
HDS Unified Storage 150125,0180.86ms1.09ms12 / 1520x
Huawei OceanStor Dorado2100 G2400,5870.60ms0.75ms0 / 1550x
Huawei OceanStor Dorado5100
600,0520.87ms1.09ms7 / 1596x

A couple of my own data points:

  • Avg latency (All utilization levels) – I just took aggregate latency of “All ASUs” for each of the utilization levels and divided it by 6 (the number of utilization levels)
  • Number of times above 1ms of latency – I just counted the number of cells in the I/O throughput table for each of the ASUs (15 cells total) that the test reported above 1ms of latency


System NameTotal
Cost per
Cost per
Usable TB
HP 3PAR 7400$148,737$0.58$133,019
HP P6500 EVA (SPC-1E)$130,982$6.55$260,239
IBM Storwize V7000$181,029$1.50$121,389
HDS Unified Storage 150$198,367$1.59$118,236
Huawei OceanStor Dorado2100 G2$227,062$0.57$61,186
Huawei OceanStor Dorado5100

Capacity Utilization

HP 3PAR 74003,250 GB1,159 GB70.46%
HP P6500 EVA (SPC-1E)1,600 GB515 GB64.41%
IBM Storwize V70003,600 GB1,546 GB84.87%
HDS Unified Storage 1503,999 GB1,717 GB85.90%
Huawei OceanStor Dorado2100 G210,002 GB3,801 GB75.97%
Huawei OceanStor Dorado510019,204 GB6,442 GB67.09%


The new utilization charts in the latest 3PAR/Huawei tests are quite nice to see, really good illustrations as to where the space is being used. They consume a full 3 pages in the executive summary. I wish SPC would go back and revise previous reports so they have these new easier forms of disclosure in them. The data is there for users to compute on their own.


This is a SPC-1e result rather than SPC-1 – I believe the work load is the same(?) they just measure power draw in addition to everything else. The stark contrast between the new 3PAR and the older P6500 is remarkable from every angle whether it is cost, performance, capacity, latency. Any way you slice it (well except power I am sure 3PAR draws more power 🙂 )

It is somewhat interesting in the power results for the P6500 that there is only a 16 watt difference between 0% load and 100% load.

I noticed that the P6500 is no longer being sold (P6550 was released to replace it – and the 3PAR 7000-series was released to replace the P6550 which is still being sold).


While I don’t expect Huawei to be a common rival for the other three outside of China perhaps, I find their configuration very curious. On the 5100 with such a large number of apparently low cost SLC(!) SSDs, and “short stroking” (even though there are no spindles I guess the term can still apply) they have managed to provide a significant amount of performance at a reasonable cost. I am confused though they claim SLC but yet they have so many disks(would think you’d need fewer with SLC), at the same time at a much lower cost. Doesn’t compute..

No software

Huawei appears to have absolutely no software options for these products – no thin provisioning, no snapshots, no replication, nothing. Usually vendors don’t include any software options as part of the testing since they are not used. In this case the options don’t appear to exist at all.

They seem to be more in line with something that LSI/NetApp E-series, or Infortrend or something like that rather than an enterprise storage system. Though looking at Infortrend’s site earlier this morning shows them supporting thin provisioning, snapshots, and replication on some arrays. Even NetApp seems to have thin provisioning on their E-series included.


3PAR’s metadata

3PAR’s utilization in this test is hampered by (relatively) excessive metadata, the utilization results say only 7% unused storage ratio which on the surface is an excellent number. But this number excludes metadata which in this case is 13%(418GB) of the system. Given the small capacity of the system this has a significant impact on utilization (compared to 3PAR’s past results). They are working to improve this.

The next largest meta data size in the above systems is IBM which has only 1GB of metadata (about 99.8% less than 3PAR). I would be surprised if 3PAR was not able to significantly slash the metadata size in the future.

3PAR 7400 SSD SPC-1 Configured Storage Capacity (one of the new charts from the SPC-1 report)

In the grand scheme of things this problem is pretty trivial. It’s not as if the meta data scales linearly with the system.

Only quad controller system

3PAR is the only SSD solution above tested with 4 controllers(totalling 4 Gen4 ASICs, 24 x 1.8Ghz Xeon CPU cores, 64GB of data cache, and 32GB of control cache), meaning with their persistent cache technology(which is included at no extra cost) you can lose a controller and keep a fully protected and mirrored write cache. I don’t believe any of the other systems are even capable of such a configuration regardless of cost.

3PAR Persistent Cache mirrors cache from a degraded controller pair to another pair in the cluster automatically.

The 7400 managed to stay below 1 millisecond response times even at maximum utilization which is quite impressive.

Thin provisioning built in

The new license model of the 3PAR 7000 series means this is the first SPC-1 result to include thin provisioning for a 3PAR system at least. I’m sure they did not use thin provisioning(no point when your driving to max utilization), but from a cost perspective it is something good to keep in mind. In the past thin provisioning would add significant costs onto a 3PAR system. I believe thin provisioning is still a separate license on the P10000-series (though would not be surprised if that changes as well).

Low cost model

They managed to do all of this while remaining a lower cost offering than the competition – the economics of this new 7000 series are remarkable.

IBM’s poor latency

IBM’s V7000 latency is really terrible relative to HDS and HP. I guess that is one reason they bought TMS.  Though it may take some time for them to integrate TMS technology (assuming they even try) to have similar software/availability capabilities as their main enterprise offerings.


With these results I believe 3PAR is showing well that they too can easily compete in the all SSD market opportunities without requiring excessive amounts of rack space or power circuits as some of their previous systems required. All of that performance(only 32 of the 48 drive bays are occupied!), in a small 4U package. Previously you’d likely be looking at a absolute minimum of half a rack!

I don’t know whether or not 3PAR will release performance results for the 7000 series on spinning rust, it’s not too important at this point though. The system architecture is distributed and they have proven time and again they can drive high utilization, so it’s just a matter of knowing the performance capacity of the controllers (which we have here), and just throwing as much disk as you want at it. The 7400 series tops out at 480 disks at the moment – even if you loaded it up with 15k spindles you wouldn’t come close to the peak performance of the controllers.

It is, of course nice to see 3PAR trouncing the primary competition in price, performance and latency. They have some work to do on utilization as mentioned above.

March 6, 2013

New Seagate Hybrid disks coming!

Filed under: Storage — Tags: , — Nate @ 10:13 am

I first saw this yesterday over on El Reg, which seemed to have found a leaked url because at the time there was no mention of the drives elsewhere on the main Seagate site. A short time ago I came across another article via Slashdot, which mentions one major thing that yesterday’s article missed: SSD accelerated write caching.

We spoke with product manager David Burks this afternoon and have new details to report, including the revelation that the latest version of the Adaptive Memory caching technology has the ability to cache some host writes.


This new, dual-mode NAND is apparently faster than the SLC flash used in the old Momentus XT.

Though I find it interesting that they do force cached writes to the spinning rust in the event of power loss, instead of letting it sit in flash to be written the next time the drive is powered up.

Experience with Momentus XT

I have been a fan of the Momentus XT for a while now, I put one in my main laptop 1-2 years ago, upgraded from the original 320GB Hitachi 7200RPM disk that it came with (Toshiba laptop carrying an Hitachi drive?? seemed strange to me), to the original 500GB XT.

The speed up was significant – even though my laptop has 8GB of memory and really never goes above 40-50% memory usage, so always I have at least 4GB available as disk buffers — the acceleration the SSD provided was very noticeable.  Though it left me wanting more … I have thought on many occasions whether or not to go full SSD – but I needed/wanted something that had a lot of space, the laptop is dual boot and I have VM images as well. With the ability to hold only a single disk internally hybrid has been the best option.

I just wish it had more flash – I’d be happy to pay much more if it had say 32GB of flash on board, instead of the 4GB that it currently has.

A few months ago I decided to upgrade my desktop at work with a pair of Momentus 750GB drives, which each have 8GB of flash on board. That system is not dual boot but does run a Windows VM 24/7 for things I need windows for (the main OS is Ubuntu 10.04 – same as my laptop). I felt that separating the I/O for the VM(s — occasionally I run other VMs for testing things locally) would be good – also isolating the flash cache so windows has it’s own and Linux gets it’s own was good too — and hell the drives were cheap.  That system has 16GB of memory so even more room for buffers – the acceleration there was even more dramatic. I had never seen Linux go from command line to X windows login screen(GDM) in a (small) fraction of a second. But it did after the cache was warmed up (significantly faster than the XT on my laptop).

The 750GB variant has four advantages over my 500GB:

  • 6G SATA (didn’t matter to me all my systems are 3G)
  • 8GB cache (double what the 500GB has)
  • “Flash Management” – whatever that is
  • “Fast Boot technology” – whatever that is

So a few weeks ago I went and bought a 750GB XT for my laptop (haven’t installed it yet), and here we have the new stuff coming out!

New 2.5″ hybrids vs old XT

There are some significant advantages of these new hybrids –

  • 1TB vs 750GB
  • $99 vs $159 (Newegg – I bought mine online at Best buy for ~$139, picked up same day!)
  • Ability to cache some writes vs. no write caching at all
  • 64MB cache vs 32MB

I did see one potentially big disadvantage of the new hybrids – power usage.  The power draw while seeking on the new hybrids(2.7W) is more than double that of the Momentus XT (1.3W). Power draw for idle is actually 0.2W less than on the XT. I wonder what drives the power usage so much higher ? Maybe it is a typo in the data sheet.

The article above reports that Seagate says there is backup power to flush the write buffers in the event of sudden power loss, a problem that seems wide spread amongst SSDs in general. Myself I had a Corsair SSD corrupt itself a little bit a few years ago when it was connected to a system with a UPS that had a dead battery. The UPS did a self test – which then cut the power to the system because the battery was dead, and the file system became corrupt. I don’t recall how I recovered the system but I think I managed to without re-installing. I thought the problem was fairly isolated to my cheap crap SSD, so was interested to learn the problem is much more wide spread covering large sectors of the market and persists even today.

Seagate of course recently announced they were discontinuing non hybrid 7200RPM laptop drives. Which is a fine idea — when you can get a 1TB hybrid drive for only $99 that’s a pretty amazing price point, even over their existing XT series.

I suspect that especially with the new price point it will cause people to think harder on whether or not they want to go full SSD on their laptops or not.

Availability of the new 2.5″ hybrids are expected in the next week or so.

New 3.5″ Desktop Hybrids

Finally there are desktop hybrids as well, which appear to be identical other than in a larger form factor, and offering a 2TB model. Prices here were reported as $99 for 1TB and $149 for 2TB.

I do have one workstation at home which I setup for some gaming about a year ago, though recently have not been using it, in it is a 750GB Momentus XT along with a few other drives – including a low end cheap 64GB Corsair SSD that I bought a few years ago. I configured Windows 7 to use that SSD as a cache(using Ready Boost – I think that is what it is called) — though have not seen any noticeable performance boost – which surprised me quite a bit(especially given the size of the SSD). I thought it would be caching the data from the games and stuff but load times still seemed relatively normal.

Availability of these 3.5″ hybrids is expected late next month.


I think I will hold onto the 750GB XT I just bought, and not return it. I don’t feel comfortable returning it just because something newer/better might be coming out. Doesn’t seem right. I’ll find a use for it somewhere.. I don’t know yet if I will upgrade my laptop with that disk, or buy the 1TB new stuff. I’ll be very interested to see the benchmarks of the new drives vs the old ones. Seagate claims the MLC is faster than the SLC used in the XT. Power draw isn’t too much of a consideration my laptop doesn’t run more than about 2.5 hours anyway(also my laptop spends 99% of it’s time plugged in), I’m not sure how much doubling the power draw on the disk(only during seeks) would have on that number.

I have read Western Digital is coming out with hybrid drives too – I wonder if they will up the ante by offering a premium model with more cache on it, I can hope at least. I’d love to see 32 or even 64GB cache in a 2.5″ form factor.

December 4, 2012

3PAR: The Next Generation

Filed under: Storage — Tags: , — Nate @ 12:40 am

(Cue Star Trek: The Next Generation theme music)

[Side note: I think this is one of my most popular post ever with nearly 3,000 hits to it so far (excluding my own IPs). Thanks for reading!]

[I get the feeling I will get lots of people linking to this since I suspect what is below will be the most complete guide as to what was released – for those of you that haven’t been here before I am in no way associated with HP or 3PAR – or compensated by them in any way of course! Just been using it for a long time and it’s one of the very few technologies that I am passionate about – I have written a ton about 3PAR over past three years]

HP felt their new storage announcements were so ground breaking that they decided to have a special event a day before HP Discover is supposed to start. They say it’s the biggest announcement for storage from HP in more than a decade.

I first got wind of what was coming last Fall, though there wasn’t much information available at the time other than a picture and some thoughts as to what might happen. Stuff wasn’t nailed down yet. I was fortunate enough to finally visit 3PAR HQ a couple of months ago and get a much more in depth briefing as to what was coming, and I’ll tell you what it’s been damn hard to contain my excitement.

HP announced a 75% year over year increase in 3PAR sales, along with more than 1,200 new customers in 2012 alone. Along with that HP said that their StoreOnce growth is 45% year over year.

By contrast HP did not reveal any growth numbers for either their Lefthand  StoreVirtual platform nor their IBRIX StoreAll platforms.

David Scott, former CEO of 3PAR tried to set the tone as a general storage product launch, they have enhancements to primary storage, to file/object scale-out storage as well as backup/archive storage.

You know I’m biased, I don’t try to hide that. But it was obvious to me at the end of the presentation this announcement was all about one thing: David’s baby – 3PAR.

Based on the web site, I believe the T-class of 3PAR systems is finally retired now. Replaced last year by the V-Class (aka P10000 or 10400 and 10800)

Biggest changes to 3PAR in at least six years

The products that are coming out today are in my opinion, the largest set of product (AND policy) enhancements/changes/etc from 3PAR in at least the past six years that I’ve been a customer.

First – a blast from the past.

The first mid range 3PAR system – the E200

Hello 2006!

There is some re-hashing of old concepts, specifically the concept of mid range. 3PAR introduced their first mid range system back in 2006, which was the system I was able to deploy – the E200. The E200 was a dual node system that went up to 4GB data cache per controller and up to 128 drives or 96TB of usable capacity whichever came first. It was powered by the same software and same second generation ASIC (code named Eagle if I remember right) that was in the high end S-class at the time.

The E200 was replaced by the F200, and the product line extended to include the first quad controller mid range system the F400 in 2009. The F-class, along with the T-class (which replaced the S-class) had the third generation ASIC in it (code named Osprey if I remember right?? maybe I have those reversed). The V-class which was released last year, along with what came out today has the 4th generation ASIC (code named Harrier).

To-date – as far as I know the F400 is still the most efficient SPC-1 result out there, with greater than 99% storage utilization – no other platforms (3PAR included) before or since have come close.

These systems, while coined mid range in the 3PAR world were still fairly costly. The main reason behind this was the 3PAR architecture itself. It is a high end architecture. Where other vendors like EMC and HDS chose radically different designs for their high end vs. their mid range, 3PAR aimed a shrink ray at their system and kept the design the same. NetApp on the other hand was an exception – they too have a single architecture that scales from the bottom on up. Though as you might expect – NetApp and 3PAR architectures aren’t remotely comparable.

Here is a diagram of the V-series controller architecture, which is very similar to the 7200 and 7400, just at a much larger scale:

3PAR V-Series ASIC/CPU/PCI/Memory Architecture

Here is a diagram of the inter-node communications on an 8-node P10800, or T800 before it, again similar to the new 7000-series just larger scale:

3PAR Cluster Architecture with low cost high speed passive backplane with point to point connections totalling 96 Gigabytes/second of throughput

Another reason for the higher costs was the capacity based licensing (& associated support). Some things were licensed per controller pair, some things based on raw capacity, some things licensed per system, etc. 3PAR licensing was not very friendly to the newbie.

Renamed Products

There was some basic name changes for 3PAR product lines:

  • The HP 3PAR InServ is now the HP 3PAR StorServ
  • The HP 3PAR V800 is now the HP 3PAR 10800
  • The HP 3PAR V400 is now the HP 3PAR 10400

The 3PAR 7000-series – mid range done right

The 3PAR 7000-series leverages all of the same tier one technology that is in the high end platform and puts it in a very affordable package, starting at roughly $25,000 for a two-node 7200 system, and $32,000 for an entry level two-node 7400 system(which can later be expanded to four nodes, non disruptively).

I’ve seen the base 7200 model (2 controllers, no disks, 3 year 24×7 4-hour on site support “parts only”) online for as low as $10,000.

HP says this puts 3PAR in a new $11 Billion market that it was previously unable to compete.

This represents roughly a 55-65% discount over the previous F-class mid range 3PAR solution. More on this later.

Note that it is not possible to upgrade in place a 7200 to a 7400. So you still have to be sure if you want a 4-node capable system to choose the 7400 up front (you can, of course purchase a two-node 7400 and add the other two nodes later).

Dual vs quad controller

The controller configurations are different between the two and the 7400 has extra cluster cross connects to unify the cluster across enclosures. The 7400 is the first 3PAR system that is not leveraging a passive backplane for all inter-node communications. I don’t know what technology 3PAR is using to provide this interconnect over a physical cable – it may be entirely proprietary. They use their own custom light weight protocols on the connection, so from a software standpoint it is their own stuff. Hardware – I don’t have that information yet.

A unique and key selling point for having a 4-node 3PAR system is persistent cache, which keeps the cache in write back mode during planned or unplanned controller maintenance.

3PAR Persistent Cache mirrors cache from a degraded controller pair to another pair in the cluster automatically.

The 3PAR 7000 series is based on what I believe is the Xyratex OneStor SP-2224 enclosure, the same one IBM uses for their V7000 StorWize system (again, speculation). Speaking of the V7000 I learned tonight that this IBM system implemented RAID 5 in software resulting in terrible performance. 3PAR RAID 5 is well – you really can’t get any faster than 3PAR RAID, that’s another topic though.

3PAR 7000 Series StorServs

3PAR 7000 Series StorServs

3PAR has managed to keep it’s yellow color, and not go to the HP beige/grey. Somewhat surprising though I’m told it’s because it helps the systems stand out in the data center.

The 7000 series comes in two flavors – a two node 7200, and a two or four node 7400. Both will be available starting December 14.

2.5″ or 3.5″ (or both)

There is also a 3.5″ drive enclosure for large capacity SAS (up to 3TB today). There are also 3.5″ SSDs but their capacities are unchanged from the 2.5″ variety – I suspect they are just 2.5″ drives in a caddy. This is based, I believe on the Xyratex OneStor SP-2424.

Xyratex OneStor SP-2424

This is a 4U, 24-drive enclosure for disks only(controllers go in the 2U chassis). 3PAR kept their system flexible by continuing to allow customers to use large capacity disks, however do keep in mind that for the best availability you do need to maintain at least two (RAID 10),  three (RAID 5), or six (RAID 6)  drive enclosures. You can forgo cage level availability if you want, but I wouldn’t recommend it – that provides an extra layer of protection from hardware faults, at basically no cost of complexity on the software side (no manual layouts of volumes etc).

HP has never supported the high density 3.5″ disk chassis on the mid range systems I believe primarily for cost, as they are custom designed. By contrast the high end systems only support the high density enclosures at this time.

3PAR High Density 3.5" Disk Chassis - not available on mid range systems

The high end chassis is designed for high availability. The disks are not directly accessible with this design. In order to replace disks the typical process is to run a software task on the array which then migrates all of the data from the disks in that particular drive sled (pack of four drives), to other disks on the system(any disks of the same RPM), once the drive sled is evacuated it can be safely removed. Another method is you can just pull the sled, the system will go into logging mode for writes for those disks(sending the writes elsewhere), and you have roughly seven minutes to do what you need to do and re-insert the sled before the system marks those drives as failed and begins the rebuild process.

The one thing that HP does not allow on SP-2424-based 3.5″ drive chassis is high performance (10 or 15K RPM) drives. So you will not be able to build a 7000-series with the same 650GB 15k RPM drives that are available on the high end 10000-series. However they do have a nice 900GB 10k RPM option in a 2.5″ form factor which I think is a good compromise.  Or you could go with a 300GB 15k RPM 2.5″. I don’t think there is a technical reason behind this, so I imagine if enough customers really want this sort of setup and yell about it, then HP will cave and start supporting it. Probably won’t be enough demand though.

Basic array specifications

72002250TB144Up to 12x8Gbps FC OR
4x8Gbps FC AND 4x10Gbps iSCSI
74004864TB480Up to 24x8Gbps FC OR
8x8Gbps FC AND 8x10Gbps iSCSI
104004800TB960Up to 96x8Gbps FC ports
Up to 16x10Gbps iSCSI
1080081600TB1920Up to 192x8Gbps FC ports
Up to 32x10Gbps iSCSI

(Note: All current 3PAR arrays have dedicated gigabit network ports on each controller for IP-based replication)

In a nut shell, vs the F-class mid range systems, the new 7000-series:

  • Doubles the data cache per controller to 12GB compared to F200, almost triple if you compare the 7400 to the F200/F400)
  • Doubles the control cache per controller to 8GB, The control cache is dedicated memory for the operating system completely isolated from the data cache.
  • Brings PCI-Express support to the 3PAR mid range allowing for 8Gbps Fibre Channel and 10Gbps iSCSI
  • Brings the mid range up to spec with the latest 4th generation ASIC, and latest Intel processor technology.
  • Nearly triples the raw capacity
  • Moves from an entirely Fibre channel based system to a SAS back end with a Fibre front end
  • Moves from exclusively 3.5″ drives to primarily 2.5″ drives with a couple 3.5″ drive options
  • Brings FC0E support to the 3PAR mid range (in 2013) for the four customers who use FCoE.
  • Cuts the size of the controllers by more than half
  • Obviously dramatically increases the I/O and throughput of the system with the new ASIC with PCIe, faster CPU cores, more CPU cores(in 7400)  and the extra cache.

Where’s the Control Cache?

Control cache is basically dedicated memory associated with the Intel processors to run the Debian Linux operating system which is the base for 3PAR’s own software layer.

HP apparently has removed all references to the control cache in the specifications, I don’t understand why. I verified with 3PAR last night that there was no re-design in that department, the separated control cache still exists, and as previously mentioned is 8GB on the 7000-series. It’s important to note that some other storage platforms share the same memory for both data and control cache and they give you a single number for how much cache there is – when in reality the data cache can be quite a bit less.

Differences between the 7200 and 7400 series controllers

Unlike previous generations of 3PAR systems, where all controllers for a given class of system were identical, the new controllers for the 104800 vs 10800, as well as the 7200 vs 7400 are fairly different.

  • 7200 has quad core 1.8Ghz CPUs, 7400 has hex core 1.8Ghz CPUs.
  • 7200 has 12GB cache/controller, 7400 has 16GB/controller.
  • 7200 supports 144 disks/controller pair, 7400 is 240 disks.
  • Along that same note 7200 supports 5 disk enclosures/pair, 7400 supports nine.
  • 7400 has extra cluster interconnects to link two enclosures together forming a mesh active cluster.

iSCSI No longer a second class citizen

3PAR has really only sort of half heartily embraced iSCSI over the years, their customer base was solidly fibre channel. When you talk to them of course they’ll say yes they do iSCSI as well as anyone else but the truth is they didn’t. They didn’t because the iSCSI HBA that they used was the 4000 series from Qlogic. The most critical failing of this part is it’s pathetic throughput. Even though it has 2x1Gbps ports, the card itself is only capable of 1Gbps of throughput. So you look at your 3PAR array and make a decision:

  • I can install a 4x4Gbps Fibre channel card and push the PCI-X bus to the limit
  • I can install a 2x1Gbps iSCSI card and hobble along with less capacity than a single fibre channel connection

I really don’t understand why they did not go back and re-visit alternative iSCSI HBA suppliers since they kept the same HBA for a whole six years. I would of liked to have seen at least a quad port 1Gbps card that could do 4Gbps of throughput. I hammered on them for years it just wasn’t a priority.

But no more! I don’t know what card they are using now, but it is PCIe and it is 10Gbps! Of course the same applies to the 10000-series – I’d assume they are using the same HBA in both but I am not certain.

Lower cost across the board for the SME

For me these details are just as much, if not more exciting than the new hardware itself. These are the sorts of details people don’t learn about until you actually get into the process of evaluating or purchasing a system.

Traditionally 3PAR has all been about margin – at one point I believe they were known to have the highest margins in the industry (pre acquisition). I don’t know where that point stands today, but from an up front standpoint they were not a cheap platform to use. I’ve always gotten a ton of value out of the platform, making the cost from my standpoint trivial to justify. But to less experienced management out there they often see cost per TB or cost per drive or support costs or whatever, compared to other platforms at a high level they often cost more. How much value you derive from those costs can very greatly.

Now it’s obvious that HP is shifting 3PAR’s strategy from something that is entirely margin focused to most likely lower margins but orders of magnitude more volume to make up for it.

I do not know if any of these apply to anything other than the 7000-series, for now assume they do not.

Thin licensing included in base software

Winning the no brainer of the year award in the storage category HP is throwing in all thin licensing as part of the  array with the base license. Prior to this there were separate charges to license thin functionality based on how much written storage was used for thin provisioning. You could license only 10TB on a 100TB array if you want, but you lose the ability to provision new thin provisioned volumes if you exceed that license (I believe there is no impact on existing volumes, but the system will pester you on a daily basis that you are in violation of the license). This approach often caught customers off guard during upgrades – they sometimes thought they only needed to buy disks – but they needed software licenses for those disks, as well as support for those software licenses.

HP finally realized that thin provisioning is the norm rather than the exception. HP is borrowing a page from the Dell Compellent handbook here.

Software License costs capped

Traditionally, most of 3PAR’s software features are based upon some measure of capacity of the system, in most cases it is raw capacity, for thin provisioning it is a more arbitrary value.

HP is once again following the Dell Compellent handbook which caps license costs at a set value(in Dell’s case I believe it is 96 spindles). For the 3PAR 7000-series the software license caps are:

  • 7200: 48 drives (33% of array capacity)
  • 7400: 168 drives (35% of array capacity)

Easy setup with Smart Start

Leveraging technology from the EVA line of arrays, HP has radically simplified the installation process of a 7000-series array, so much so that the customer can now perform the installation on their own without professional services. This is huge for this market segment. The up front professional services to install a mid range F200 storage system had a list price of $10,000 (as of last year anyway).

User serviceable components

Again for the first time in 3PAR’s history a customer will be allowed to replace their own components (disks at least, I assume controllers as well though). This again is huge – it will slash the entry level pricing for support for organizations that have local support staff available.

The 7000-series comes by default with a 24x7x365 4-hour on site support (parts only). I believe software support and higher end on site services are available for an additional charge.

All SSD 7000 series

Like the 10000-series, the 7000-series can run on 100% SSDs, a configuration that for some reason was not possible on the previous F-series of midrange systems (also I think T-class could not as well).

HP claims that with a maximum configuration, a 4-node 7400 maxed out with 240 x 100 or 200GB SSDs the system can achieve 320,000 IOPS, a number which HP claims is a 2.4x performance advantage to their closest priced competitor. This number is based on a 100% random read test with 8kB block sizes @ 1.6 milliseconds of latency. SPC-1 numbers are coming – I’d guesstimate that SPC-1 for the 7400 will be in the ~110,000 IOPS range since it’s roughly 1/4th the power of a 10800 (half the nodes, and each node has half the ASICs & CPUs and far less data cache).

HP is also announcing their intention to develop a purpose built all-SSD solution based on 3PAR technology.

Other software announcements

Most of them from here.

Priority Optimization

For a long time 3PAR has touted it’s ability to handle many workloads of different types simultaneously, providing multiple levels of QoS on a single array. This was true, to a point.

3PAR: Mixed quality of service in the same array

While it is true that you can provide different levels of QoS on the same system, 3PAR customers such as myself realized years ago that it could be better. A workload has the potential to blow out the caches on the controllers (my biggest performance headache with 3PAR – it doesn’t happen often, all things considered I’d say it’s probably a minor issue compared to competing platforms but for me it’s a pain!). This is even more risky in a larger service provider environment where the operator has no idea what kind of workloads the customers will be running. Sure you can do funky things like carve the system up so less of it is impacted when that sort of event happens but there are trade offs there as well.

Priority Optimization

The 3PAR world is changing – with Priority Optimization – a feature that essentially beta at this point, allows the operator to set thresholds both on an IOPS as well as bandwidth perspective. The system reacts basically in real time. Now on a 3PAR platform you can guarantee a certain level of performance to a workload. Whereas in the past, there was a lot more hope involved.  Correct me if I’m wrong but I thought this sort of QoS was exactly the sort of thing that Oracle Pillar used to tout. I’m not sure if they had knobs like this, but I do recall them touting QoS a lot.

Priority Optimization will be available sometime in 2013 – I’d imagine it’d be early 2013 but not sure.

Autonomic Replication

As I’ve said before – I’ve never used 3PAR replication – never needed it. I’ve tended to build things so that data is replicated via other means, and low level volume-based replication is just overkill – not to mention the software licensing costs.

3PAR Synchronous long distance replication: unique in the mid range

But many others I’m sure do use it, and this industry first as HP called it is pretty neat. Once you have your arrays connected, and your replication policies defined, when you create a new volume on the source array, all details revolving around replication are automatically configured to protect that volume according to the policy that is defined. 3PAR replication was already a breeze to configure, this just made it that much easier.

Autonomic Rebalance

3PAR has long had the ability to re-stripe data across all spindles when new disks were added, however this was always somewhat of a manual process, and it could take a not insignificant amount of time because your basically reading and re-writing every bit of data on the system. It was a very brute force approach. On top of that you had to have a software license for Dynamic Optimization in order to use it.

Autonomic rebalance is now included in the base software license and will automatically re-balance the system when resources change, new disks, new controllers etc. It will try, whenever possible, to move the least amount of data – so the brute force approach is gone, the system has the ability to be more intelligent about re-laying out data.

I believe this approach also came from the EVA storage platform.

Persistent Ports

This is a really cool feature as well – it gives the ability to provide redundant connectivity to multiple controllers on a 3PAR array without having to have host-based multipathing software. How is this possible? Basically it is NPIV for the array. Peer controllers can assume the world wide names for the ports on their partner controller. If a controller goes down, it’s peer assumes the identities of that controller’s ports, instantaneously providing connectivity for hosts that were (not directly) connected to the ports on the downed controller. This eliminates pauses for MPIO software to detect faults and fail over, and generally makes life a better place.

HP claims that some other tier 1 vendors can provide this functionality for software changes, but they do not today, provide it for hardware changes. 3PAR provides this technology for both hardware and software changes – on all of their currently shipping systems!

Peer Persistence

This is basically a pair of 3PAR arrays acting as a transparent fail over cluster for local or metro distances. From the PDF

The Peer Persistence software achieves this key enhancement by taking advantage of the Asymmetric Logical Unit Access (ALUA) capability that allows paths to a SCSI device to be marked as having different characteristics.

Peer persistence also allows for active-active to maximize available storage I/O under normal conditions.

Initially Peer Persistence is available for VMware, other platforms to follow.

3PAR Peer Persistence

Virtualized Service Processor

All 3PAR systems have come with a dedicated server known as the Service Processor, this acts as a proxy of sorts between the array and 3PAR support. It is used for alerting as well as remote administration. The hardware configuration of this server was quite inflexible and it made it needlessly complex to deploy in some scenarios (mainly due to having only a single network port).

The service processor was also rated to consume a mind boggling 300W of power (it may of been a legacy typo but that’s the number that was given in the specs).

The Service processor can now be deployed as a virtual machine!

Web Services API

3PAR has long had a CIM API (never really knew what that was to be honest), and it had a very easy-to-use CLI as well (used that tons!), but now they’ll have a RESTful Web Services API that uses JSON (ugh, I hate JSON as you might recall! If it’s not friends with grep or sed it’s not friends with me!). Fortunately for people like me we can keep using the CLI.

This API is, of course, designed to be integrated with other provisioning systems, whether it’s something off the shelf like OpenStack, or custom stuff organizations write on their own.

Additional levels of RAID 6

3PAR first introduced RAID 6 (aka RAID DP) with the aforementioned last major software release three years ago, with that version there were two options for RAID 6:

  • 6+2
  • 14+2

The new software adds several more options:

  • 4+2
  • 8+2
  • 10+2

Thick Conversion

I’m sure many customers have wanted this over the years as well. The new software will allow you to convert a thin volume to a thick (fat) volume. The main purpose of this of course is to save on licensing for thin provisioning when you have a volume that is fully provisioned (along with the likelihood of space reclamation on that volume being low as well). I know I could of used this years ago.. I always shook my fist at 3PAR when they made it easy to convert to thin, but really impossible to convert back to thick (without service disruption anyway). Basically all that is needed is to flip a bit in the OS (I’m sure the nitty gritty is more complicated).

Online Import

This basically allows EVA customers to migrate to 3PAR storage without disruption (in most cases).

System Tuner now included by default

The System Tuner package is now included in the base operating system (at least on 7000-series). System Tuner is a pretty neat little tool written many years ago that can look at a 3PAR system in real time, and based on thresholds that you define recommend dynamic movement of data around the system to optimize the data layout. From what I recall it was written in response to a particular big customer request to prove that they could do such data movement.

3PAR System Tuner moves chunklets around in real time

It is important to note that this tool is an on demand tool, when running it gathers tens of thousands of additional performance statistics from the chunklets on the system. It’s not something that can(or should be) run all the time. You need to run it when the workload you want to analyse is running in order to see if further chunklet optimization would benefit you.

System Tuner will maintain all existing availability policies automatically.

In the vast majority of cases the use of this tool is not required. In fact in my experience going back six years I’ve used it on a few different occasions, and in all cases it didn’t provide any benefit. The system generally does a very good job of distributing resources. But if your data access patterns change significantly, System Tuner may be for you – and now it’s included!

3PAR File Services

This announcement was terribly confusing to me at first. But I got some clarification. The file services module is based on the HP StoreEasy 3830 storage gateway.

  • Hardware platform is a DL380p Gen8 rack server attached to the 3PAR via Fibre Channel
  • Software platform is Microsoft Windows Storage Server 2012 Standard Edition
  • Provides NFS, CIFS for files and iSCSI for block
  • SMB 3.0 supported (I guess that is new, I don’t use CIFS much)
  • NFS 4.1 supported (I’ll stick to NFSv3, thanks – I assume that is supported as well)
  • Volumes up to 16TB in size
  • Integrated de-duplication (2:1 – 20:1)
  • VSS Integration – I believe that means no file system-based snapshots (e.g. transparent access of the snapshot from within the same volume) ?
  • Uses Microsoft clustering for optional HA
  • Other “Windowsey” things

The confusion comes from them putting this device under the 3PAR brand. It doesn’t take a rocket scientist to look at the spec sheets and see there are no Ethernet ports on the arrays for file serving. I’d be curious to find out the cost of this file services add-on myself, and what it’s user interface is like. I don’t believe there is any special integration between this file services module and 3PAR – it’s just a generic gateway appliance.

For someone with primarily a Linux background I have to admit I wouldn’t feel comfortable relying on a Microsoft implementation of NFS for my Linux boxes (by the same token I feel the same way about using Samba for serious Windows work – these days I wouldn’t consider it – I’d only use it for light duty simple stuff).

Oh while your at it HP – gimme a VSA of this thing too.

Good-bye EVA and VSP, I never knew thee

Today I think was one of the last nails in the coffin for EVA. Nowhere was EVA present on the presentation other than providing tools to seamlessly migrate off of EVA onto 3PAR. Well that and they have pulled some of the ease of use from EVA into 3PAR.

Literally nowhere was Hitachi VSP (aka HP P9500). Since HP acquired 3PAR the OEM’d Hitachi equipment has been somewhat of a fifth wheel in the HP storage portfolio. Like the EVA, HP had customers who wanted the VSP for things that 3PAR simply could not or would not do at the time. Whether it was mainframe connectivity, or perhaps ultra high speed data warehousing. When HP acquired 3PAR, the high end was still PCI-X based and there wasn’t a prayer it was going to be able to dish out 10+ GB/second. The V800 changed that though. HP is finally making inroads into P9500 customers with the new 3PAR gear. I personally know of two shops that have massive deployments of HP P9500 that will soon have their first 3PAR in their respective data centers. I’m sure many more will follow.

Time will tell how long P9500 sticks around, but I’d be shocked – really shocked if HP decided to OEM whatever came next out of Hitachi.

What’s Missing

This is a massive set of announcements, the result of blood sweat and tears of many engineers work, assuming it all works as advertised they did an awesome job!


There’s always a BUT isn’t there.

There is one area that I have hammered on 3PAR for what feels like three years now and haven’t gotten anywhere, the second area is more of a question/clarification.

SSD-Accelerated write caching

Repeat after me – AO (Adaptive Optimization) is not enough. Sub LUN auto tiering is not enough. I brought this up with David Scott himself last year, and I bring it up every time I talk to 3PAR. Please, I beg you please, come out with SSD-accelerated write caching technology. The last time I saw 3PAR in person I gave them two examples – EMC FastCache which is both a read and a write back cache. The second is Dell Compellent’s Data Progression technology. I’ve known about Compellent’s storage technology for years but there was one bit of information that I was not made aware of until earlier this year. That is their Data Progression technology by default automatically sends ALL writes (regardless of what tier the blocks live on), to the highest tier. On top of that, this feature is included in the base software license, it is not part of the add-on automatic tiering software.

The key is accelerating writes. Not reads, though reads are nice too. Reads are easy to accelerate compared to writes. The workload on my 3PAR here at my small company is roughly 92% write (yes you read that right). Accelerating reads on the 3PAR end of things won’t do anything for me!

If they can manage to pull themselves together and create a stable product, the Mt. Rainier technology from Qlogic could be a stop gap. I believe NetApp is partnered with them already for those products. Mt. Rainier, other than being a mountain near Seattle, is a host-based read and write acceleration technology for fibre channel storage systems.

Automated Peer Motion

HP released this more than a year ago – however to-date I have not noticed anything revolving around automatic movement of volumes. Call it what you want, load balancing, tiering, or something, as far as I know at this point any actions involving peer motion are entirely manual. Another point is I’m not sure how many peers an array can have. HP tries to say it’s near limitless – could you have 10 ? 20 ? 30 ? 100 ?  I don’t know the answer to that.

Again going back to Dell Compellent (sorry) their Live Volume software has automatic workload distribution. I asked HP about this last year and they said it was not in place then – I don’t see it in place yet.

That said – especially with the announcements here I’m doubling down on my 3PAR passion. I was seriously pushing Compellent earlier in the year(one of the main drivers was cost – one reseller I know calls them the Poor Man’s 3PAR) but where things stand now, their platform isn’t competitive enough at this point, from either a cost or architecture standpoint. I’d love to have my writes going to SSD as Compellent’s Data Progression does things, but now that the cost situation is reversed, it’s a no brainer to stick with 3PAR.

More Explosions

HP needs to take an excursion and blow up some 3PAR storage to see how fast and well it handles disaster recovery, take that new Peer Persistence technology and use it in the test.

Other storage announcements

As is obvious by now, the rest of the announcements pale in comparison to what came out of 3PAR. This really is the first major feature release of 3PAR software in three years (the last one being 2.3.1 which my company at the time participated in the press event and I was lucky enough to be the first production customer to run it in early January 2010 (had to for Exanet support – Exanet was going bust and I wanted to get on their latest code before they went *poof*)).

StoreOnce Improvements

The StoreOnce product line was refreshed earlier in the year and HP made some controversial performance claims. From what I see the only improvement here is they brought down some performance enhancements from the high end to all other levels of the StoreOnce portfolio.

I would really like to see HP release a VMware VSA with StoreOnce, really sounds like a no brainer, I’ll keep waiting..

StoreAll Improvements

StoreAll is the new name for the IBRIX product line, HP’s file and object storage offering. The main improvement here is something called Express Query which I think is basically a meta data search engine that is 1000s of times faster than using regular search functions for unstructured data. For me I’d rather just structure the data a bit more, the example given is tagging all files for a particular movie  to make it easier to retrieve later. I’d just have a directory tree and put all the files in the tree – I like to be organized. I think this new query tool depends on some level of structure – the structure being the tags you can put on files/objects in the system.

HP Converged storage growth - 38% YoY - notice no mention of StoreAll/IBRIX! Also no mention of growth for Lefthand either

HP has never really talked a whole lot about IBRIX – and as time goes on I’m understanding why. Honestly it’s not in the same league (or sport for that matter) for quality and reliability as 3PAR is, not even close. It lacks features, and according to someone I know who has more than a PB on HP IBRIX storage (wasn’t his idea it’s a big company)  it’s really not pleasant to use. I could say more but I’ll end by saying it’s too bad that HP does not have a stronger NAS offering. IBRIX may scale well on paper, but there’s a lot more to it than the paper specs of course. I went over the IBRIX+3PAR implementation guide, for using 3PAR back end storage on a IBRIX system and wasn’t impressed with some of the limitations.

Like everything else, I would like to see a full IBRIX cluster product deployable as a VMware VSA. It would be especially handy for small deployments(e.g. sub 1TB). The key here is the high availability.

HP also announced integration between StoreAll ExpressQuery and Autonomy software. When the Autonomy guy came on the stage I really just had one word to describe it: AWKWARD – given what happened recently obviously!


This was known as the P4000, or Lefthand before that. It was also refreshed earlier in the year. Nothing new announced today. HP is trying to claim the P4000 VSA as Software Defined Storage (ugh).


Make no mistake people – this storage announcement was all about 3PAR. David Scott tried his best to share the love, but there just wasn’t much exciting to talk about outside of 3PAR.

6,000+ words ! Woohoo. That took a lot of time to write, hopefully it’s the most in depth review of what is coming out.

December 3, 2012

The final countdown

Filed under: Storage — Tags: — Nate @ 6:31 am


It’s 5:30 AM ..

I got paged this morning for something someone else broke, so I was up already, I know HP was going to announce something soon (was expecting tomorrow, HP Discover is Dec 4-6th and it’s still December 3 in Germany), but it seems like it is today instead, and as I write this we’re about 30-minutes away.

I’m not all sure what is being announced vs what I have learned already, so am excited to see this news will finally get released.

UPDATE – It seems the new 7200 and 7400 arrays have been announced, waiting to see if there is more or not. Entry level pricing for 3PAR just got cut by about 2/3rds to about $20-25,000 with the introduction of these arrays in the mid range. There’s a bunch more though, once I get more clarification as to what I can talk about then I’ll have something else to write..


October 9, 2012

Backblaze’s answer to the Thai flooding

Filed under: Storage — Tags: , — Nate @ 10:37 am

Saw an interesting article over at Slashdot, then went to GigaOm, and then went to the source. Aside from the sick feeling I felt when a cloud storage provider is sourcing their equipment through Costco, or Newegg, the more interesting aspect to the Backblaze story, that I wasn’t aware of before is the people in the Slashdot thread pointing out the limitations to their platform.

Here I was thinking Backblaze is a cheap way to store stuff off site but its a lot more complicated than that. For my own off site backups I use my own hardware hosted in a co-location facility that is nearby. The cost is reasonable considering the flexibility I have (it seems far cheaper than any cloud storage I have come across anyways which honestly surprises me given I have no leverage to buy hardware).

Anyways back to Backblaze, the model really sort of reminds me of the model that so many people complain about when it comes to broadband and wireless data usage plans. The price is really cheap – they did that part well.

The most startling thing to me is they delete data 30 days after you delete it locally. They don’t allow storage of many common types of files like ISO images, virtual machine images. They acknowledge

Backblaze is not designed as an additional storage system when you run out of space.

(having wrote this I could just as easily see consumer internet companies saying that they are not designed to replace your Cable/Satellite with Netflix+Hulu)

At the same time they advertise unlimited storage. They talk about how much cheaper they are than (shudder) Amazon, and other providers (as well as doing things in house), but don’t mention these key comparison points. I believe one of the posts on slashdot even claimed that Backblaze goes out of their way to detect network drives and perhaps iSCSI attached network storage and blocks that from being backed up as well.

For all I know the other players in the space have similar terms, I haven’t investigated, I was just kind of surprised to see such mixed messages coming from them, from one side they say unlimited storage for a cheap rate, while at the same time they put all sorts of restrictions on it.

The up side is of course they seem to be fairly up front about what they limit when you dig more into the details, but at the same time the broadband and wireless data providers are fairly upfront as well, but that doesn’t stop people from complaining at ever increasing volumes.

I’d think they could do a lot more if they expanded the scope of their support with tiers of service, For example extending the window of storage from 30 days to some arbitrary longer period, for some marginal increase in cost. But maybe not.

I’m sure they run a fine service for the target market, I was always sort of curious how they managed the cost model, outside of the hardware anyways, reading this today really enlightened me as to how that strategy works.

Learn something new every day (almost).

October 5, 2012

HP Releases new 3PAR vCenter Plugin

Filed under: Storage — Tags: , — Nate @ 3:31 pm

[I haven’t written a story about 3PAR in the past five minutes so I suppose I’m due..]

Well it’s not that new, to be honest I don’t know how old it is(maybe it’s 6 months old!). I was complaining to 3PAR recently about the lack of functionality in their vCenter plugin and was told that they had a newer version that had some good stuff in it.

The only caveat is this version couldn’t be downloaded from the HP website (no versions can, I looked as recently as yesterday afternoon). It’s only available in the media kit, aka CDROM. I didn’t remember which version was the newer one and when I was told about the newer one I didn’t know which version I had. So I asked the Seattle account team what the current version is because the version I was handed with our array which was installed in December was 2.2.0. It had some marginal improvements in the VMware Recovery Manager (I don’t need the recovery manager), but the vCenter plugin itself was sorely lacking, it felt like it had gone nowhere since it was first released what seems like three years ago (maybe it was two).

I track 3PAR pretty closely as you might imagine, and if I had absolutely no idea there was a new version then I suspect there are a lot of customers out there that have no idea. I never noticed any notifications, there’s no “upgrade checker” on the software side etc.

Anyways, sure enough they get back to me and say 2.2.3 is the latest and sent me a electronic copy of the ISO, and I installed it. I can’t say it’s massively better but it does address two basic sets of functionality that was lacking previously:

  • Ability to cache user credentials to the array in vCenter itself (before you had to re-login to the array every time you loaded the vCenter client)
  • Ability to provision storage from vCenter (tried this – it said I had to configure a storage template before it would function – I’ve never needed templates on 3PAR before so not sure why i do now – I suppose it just makes it more simple, though it’d be nice if there was an advanced check box to continue without a template)

There may be other things too that I haven’t noticed. I don’t think it is top notch yet, I’m fairly certain both EMC and NetApp’s integration packages are much more in depth. Though it wouldn’t surprise me if 3PAR now has the resources to fix the situation on their end, client side software was never really a strong point of theirs. For all I know they are busy re-writing it in a better language – to run on the new vCenter web console.

HP 3PAR vCenter Plugin

Based on the UI, I didn’t get the impression that the plugin could export storage to the whole cluster, since the provision storage option was available under each server but wasn’t visible in the cluster. But who knows, maybe if I took the time to make a template I’d see that it could export to the whole cluster at once..

Not that I needed to provision storage from vCenter, for me it’s much simpler to just ssh in and do it –

  • Create the volume of whatever size I want
  • Export the volume to the cluster (all servers with 1 command)

It really is just two commands. Well three if you count the ssh command line itself to login to the system. I can see the value for less technical folks though so I think it’s important functionality to have. I can accomplish that in a fraction of the amount of time it takes me to login to vCenter, fire up silver light and go through a wizard.

Something I have wanted to see is more integration from the performance monitoring/management standpoint. I don’t know what all hooks are available in vCenter for this sort of thing.

The 3PAR plugin is built using Microsoft Silverlight which was another thorn in my side earlier this year – because Silverlight did not support 64-bit windows. So I couldn’t run the plugin from the vCenter server itself (normally I just remote desktop to the vCenter server and run the client locally – the latency running it over the WAN can get annoying). But to my surprise Microsoft released an update at some point in the past several months and Silverlight now works in 64bit!

So if you happen to want this newer version of software (the plugin is free), contact your HP account team or file a support ticket to get it. Be sure to tell them to make that available for download, there’s no reason to not make it available to download. The VMware Recovery Manager is not free by contrast (both are distributed together), however the Recovery manager checks the license status on the array, so you can install it, but it won’t work unless the array has the license key.

On a somewhat related note I installed a Qlogic management plugin in vCenter a couple of months back, among other things it allows you to upgrade the firmware of their cards from vCenter itself. The plugin isn’t really high quality though, the documentation is poor and it was not too easy to get up and going(unlike the 3PAR plugin the Qlogic plugin cannot be installed on the vCenter server – I tried a dozen times). But it is sort of neat to see what it has, it shows all of the NICs and HBAs and what they are connected to. I think I have so many paths and connections that it seems to make the plugin go unresponsive and hang the vCenter client much of the time (eventually it unfreezes). Because of that I have not trusted it to do firmware upgrades.

Qlogic vCenter Plugin

The Qlogic plugin requires software to be installed on each physical server that you want Qlogic information for(which also requires a reboot). The host software, from what I remember, is also not compatible with VMware Update Manager, so at least I had to install it from the CLI. You can download the Qlogic plugin from their website, here is one link.

Both plugins need a lot of work, Qlogic’s is pretty much unusable, I have a small environment here and it’s dog slow. 3PAR’s well it is more usable now, performance is fine, and at least the two new features above bring it out of the unusable territory for myself (I probably still won’t use it but it provides at least some value now for less technical folks where before it did not).

August 24, 2012

3PAR: Helping and Hurting HP ?

Filed under: Storage — Tags: — Nate @ 8:59 am

Here I go, another blog post starting with question. Yet another post sparking speculation on my part due to an article by our friends at The Register who were kind enough to do some number crunching of HP’s latest quarterly numbers were storage revenues were down about 5%.

Apparently a big chunk of the downward slide for revenues was declines in EVA and tape, offset to some degree by 3PAR and StoreOnce de-dupe products.

I suppose this thought could apply to both scenarios, but I’ll focus on the disk end, since I have no background on StoreOnce.

Before HP acquired 3PAR, obviously EVA was a juicy target to go after to replace EVAs with 3PARs. The pitch was certainly you can get a hell of a lot more done on less 3PAR than you can with more EVA. So you’ll end up saving money. I’ve never used EVA before myself, heard some good aspects of it and some really bad aspects of it, I don’t think I’d ever want to use EVA regardless.

I am sure that 3PAR reps (those that haven’t left anyways – I’ve heard from numerous sources they outclass their HP counterparts by leagues and leagues), who are now responsible for pitching HP’s entire portfolio obvious have a strong existing bias towards 3PAR and away from the other HP products. They try to keep a balanced viewpoint but I’m sure that’s hard to do, especially after they’ve been spending so much time telling the world how much these other products are bad and why the customer should use 3PAR instead. Can’t blame them, its a tough switch to make.

So, assuming you can get a hell of a lot more done on a smaller/fewer 3PAR system(s) than EVA  – which I think is totally true, (with some caveat as to what sort of discounts some may be able to score on EVA, 3PAR has traditionally had some strict margin rules where they have no problem walking away from a deal if the margin is too low), add to that the general bias of at least part of the sales force, as well as HP’s general promotion that 3PAR is the future, and you can quite possibly get lower overall revenue while the customers are saving money by having to buy fewer array resources to accomplish the same (or more) tasks.

3PAR revenue was up more than 60% apparently, on top of the previous gains made since the acquisition.

It would be very interesting to me to see how much consolidation some of these deals end up being – traditionally NetApp I think has been the easiest target for 3PAR, I’ve seen some absolutely massive consolidation done in the past with those products, it was almost comical in some cases. I bet EVA is similar.

Now the downside to the lower revenues, and I’ve seen this at both Dell and HP – both companies are feeling tremendous pressure to try to outperform, they haven’t been able to do it on the revenue side, so they’ve been squeezing on internal costs, which really can degrade services. Overall quality of the sales forces at the likes of HP and Dell have traditionally been terrible, compared to the smaller company counterparts (at least in storage). Add to that the internal politics and region limitations that the companies place on their sales forces further complicates and frustrates the quality people internally as well as customers externally. Myself I was unable to get anything out of a local HP/3PAR account team for months in the Bay Area, so I reached out to my friends in Seattle and they turned some stuff around for me in a matter of hours no questions asked, and they didn’t get any credit (from HP) for it either. Really sad situation for both sides.

I don’t have much hope that HP will be willing or able to retain the top quality 3PAR folks at least on the sales side over the medium term, they, like Dell seem focused on driving down costs rather than keeping quality high, which is a double edged sword. The back end folks will probably stick around for longer, given that 3PAR is one of the crown jewels in HP’s enterprise portfolio.

For some reason I’m immediately reminded of this quote from Office Space:

[..] "that is not right, Michael. For five years now, you've worked your ass  off at Initech, hoping for a promotion or some kind of profit sharing  or something. Five years of your mid-20s now, gone. And you're gonna go  in tomorrow and they're gonna throw you out into the street. You know  why? So Bill Lumbergh's stock will go up a quarter of a point."


One of 3PAR’s weak points has been at the low end of the market, say sub $100k deals, is a space 3PAR has never tried to compete in.  Apparently according to The Register the HP P4000/Lefthand side of things is not doing so hot, and also seemed to be HP’s go-to product for this price range. This product range is what HP used to be excited about, before 3PAR, I attended a storage briefing at a VMware User group meeting just before HP bought 3PAR, expecting some sort of broad storage overview, but it was entirely Lefthand focused. While Lefthand has some interesting tech (the network RAID is pretty neat), for the most part I’d rather pay more and use 3PAR obviously.

I wonder what will happen to Lefthand in the future, will the best of it’s tech get rolled up into 3PAR? or vise versa? Or maybe it will just stay where it’s at, the one good thing Lefthand has is the VSA, it’s not as complete as I’d like to see it, but it’s one of the very few VSAs out there.

Dell has been busy trying to integrate their various storage acquisitions whether it’s Compellent, Ocarnia, Exanet, and I think there was one or two more that I don’t remember. Storage revenues there down as well. I’m not sure how much of the decline has to do with Dell terminating the EMC reselling stuff at this point, but it seems like a likely contributor to the declines in their case.

August 23, 2012

Real time storage auto tiering?

Filed under: Storage — Tags: — Nate @ 9:42 am

Was reading this article over by our friends at The Register, and apparently there is some new system from DotHill that claims to provide real time storage tiering – that is moving data between tiers every 5 seconds.

The AssuredSAN Pro 5000 has a pretty bad name, borrows the autonomic term that 3PAR loves to use so much(I was never too fond of it, preferred the more generic term automatic), in describing their real time tiering. According to their PDF, they move data around in 4MB increments, far below most other array makers.

From the PDF –

  • Scoring to maintain a current page ranking on each and every I/O using an efficient process that adds less than one microsecond of overhead. The algorithm takes into account both the frequency and recency of access. For example, a page that has been accessed 5 times in the last 100 seconds would get a high score.
  •  Scanning for all high-scoring pages occurs every 5 seconds, utilizing less than 1.0% of the system’s CPU. Those pages with the highest scores then become candidates for promotion to the higher-performing SSD tier.
  •  Sorting is the process that actually moves or migrates the pages: high scoring pages from HDD to SSD; low scoring pages from SDD back to HDD. Less than 80 MB of data are moved during any 5 second sort to have minimal impact on overall system performance.

80MB of data every 5 seconds is quite a bit for a small storage system, I have heard of situations where auto tiering has such an impact that it actually made things worse due to so much data moving around internally on the system, and had to be disabled. I would hope they have some other safe gaurds in there like watching spindle latency and scheduling the movements to be low priority, and perhaps even canceling movements if the job can’t be completed in some period of time.

The long delay time between data movements is perhaps the #1 reason why I have yet to buy into the whole automagic storage tiering concept. My primary source of storage bottlenecks over the years has been some out-of-nowhere job that generates a massive amount of writes and blows out the caches on the system. Sometimes the number of I/Os inbound is really small too, but the I/O size can be really big (5-10x normal) and is then split up into smaller I/Os on the back end and the IOPS are multiplied as a result. The worst offender was some special application I supported a few years ago which would, as part of it’s daily process dump 10s of GB of data from dozens of servers in parallel to the storage system as fast as it could. This didn’t end well as you can probably imagine, at the peak we had about roughly 60GB of RAM cache between the SAN and NAS controllers. I tried to get them to re-architect the app to use something like local Fusion IO storage but they did not they needed shared NFS. I suspect this sort of process would not be helped too much by automatic storage tiering because I’m sure the blocks are changing each day to some degree. This is also why I have not been a fan of things like SSD read caches (hey there NetApp!), and of course having a SSD-accelerated write cache on the server end doesn’t make a lot of sense either since you could lose data in the event the server fails. Unless you have some sort of fancy mirroring system, to mirror the writes to another server but that sounds complicated and problematic I suspect.

Compellent does have one type of tiering that is real time though – all writes by default go to the highest tier and then are moved down to lower tiers later. This particular tiering is even included in the base software license. It’s a feature I wish 3PAR had.

This DotHill system also supports thin provisioning, though no mention of thin reclaiming, not too surprising given the market this is aimed at.

They also claim to have some sort of rapid rebuild, by striping volumes over multiple RAID sets, I suppose this is less common in the markets they serve (certainly isn’t possible on some of the DotHill models), this of course has been the norm for a decade or more on larger systems. Rapid rebuild to me obviously involves sub disk distributed RAID.

Given that DotHill is a brand that others frequently OEM I wonder if this particular tech will bubble up under another name, or will the bigger names pass on this in the fear it may eat into their own storage system sales.

« Newer PostsOlder Posts »

Powered by WordPress