TechOpsGuys.com Diggin' technology every day

28Oct/10Off

Compellent beats expectations

TechOps Guy: Nate

Earlier in the year Compellent's stock price took a big hit following lower expectations for sales and stuff, a bunch of legal stuff followed that, it seems yesterday they redeemed themselves though with their stock going up nearly 33% after they tripled their profits or something.

I've had my eye on Compellent for a couple of years now, don't remember where I first heard about them. They have similar technology to 3PAR, just it's implemented entirely in software using Intel CPUs as far as I know vs 3PAR leveraging ASICs (3PAR has Intel CPUs too but they aren't used for too much).

I have heard field reports that because of this that their performance is much more on the lower end of things, they have never published a SPC-1 result and I don't know anyone that uses them so don't know how they really perform.

They seem to use the same Xyratex enclosures that most everyone else uses. Compellent's controllers do seem to be somewhat on the low end of things, I really have nothing other to go on other than cache. With their high end controller coming in at only 3.5GB of cache (I assume 7GB mirrored for a pair of controllers?) it is very light on cache. The high end has a dual core 3.0Ghz CPU.

The lower amount of cache combined with their software-only design and only two CPUs per controller and the complex automated data movement make me think the systems are built for the lower end and not as scalable, but I'm sure perfectly useful for the market they are in.

Would be nice to see how/if their software can scale if they were to put say a pair of 8 or 12 core CPUs in their controllers. After all since they are leveraging x86 technology performing such an upgrade should be pretty painless! Their controller specs have remained the same for a while now(as far back as I can remember). The bigger CPUs will use more power, but from a storage perspective I'm happy to give a few hundred more watts if I can get 5x+ the performance, don't have to think once, yet alone twice.

They were, I believe the first to have automagic storage tiering and for that they deserve big props, though again no performance numbers posted (that I am aware of) that can illustrate the benefits this technology can bring to the table. I mean if anybody can prove this strategy works it should be them right? On paper it certainly sounds really nice but in practice I don't know, haven't seen indications that it's as ready as the marketing makes it out to be.

My biggest issue with automagic storage tiering is how fast the array can respond to "hot" blocks and optimize itself, which is why I think from a conceptual perspective I really like the EMC Fast Cache approach more (they do have FAST LUN and sub LUN tiering too). Not that I have any interest in using EMC stuff but they do have cool bits here and there.

Maybe Compellent the next to get bought out (as a block storage company yeah I know they have their zNAS), I believe from a technology standpoint they are in a stronger position than the likes of Pillar or Xiotech.

Anyway that's my random thought of the day

26Oct/10Off

Rust in peace

TechOps Guy: Nate

So one of of the former TechOpsGuys Tycen emailed our little team of people yesterday mentioning that the company where we all met (Dave, Jason, Nate & Tycen) is now dead and buried according to an article on TechFlash. We were all on the same operations team in case it wasn't obvious.

Recruiting.com, the Seattle online recruiting startup formerly known as Jobster, has quietly been sold in a deal of unknown size to Phoenix, Arizona-based Jobing, TechFlash has learned. The acquisition, which closed in July, marks the final chapter for one of Seattle's most heavily funded Internet companies of the past decade. Founded in 2004 by Internet entrepreneur Jason Goldberg, Recruiting.com raised $55 million from a slate of investors that included Ignition Partners, Reed Elsevier Ventures, Trinity Ventures, Mayfield Fund and others.

Where did the money go? I don't know, we were really efficient in IT, and I think development was pretty good, so I can only think that it was the less technical areas of the company that spent most of the cash. $55 million doesn't seem like enough to say "one of the most heavily funded". Doesn't seem like much at all. I know it was a different era but I remember at Freeinternet.com (headquartered in Federal Way, near Seattle) they were spending $7 million/month on bandwidth they weren't even using (they didn't know they had it). That is just one of the little factoids.

The strategies the company had while I was there were typical? Nothing short of their original strategy made any sense to me and unfortunately for them they kept pulling resources out of that original strategy (the one generating the majority of the revenue but it wasn't shiny). We knew we were in trouble when we were denied the right to remove email addresses from our databases that were bouncing back causing us to get blacklisted on mail servers due to excessive bouncing. It would hurt our user count. Well those users don't exist anymore!!

Jobster certainly provided me with several opportunities to round out some of my skills and experience. I had learned a lot prior to that but often worked with dedicated people whether it was networking people, storage people etc. So while I knew what I was doing it was nice to be able to build things up from scratch. Did a data center move while I was there, moving out of a facility that was plagued with power outages in cabinets spread out all over the place to a better facility in a consolidated cage. Got a lot more hands on with our Oracle Databases as well, at previous companies we had dedicated Oracle folks and they pretty much ran everything. Speaking of Oracle even transitioned the company from Enterprise Edition to Standard Edition which of course saved a ton of greenbacks.

I went looking for my stock certificates I bought a few shares after I left, the price was cheap (~$0.15) just in case. For some reason I could not find them, maybe they didn't send them to me I don't remember it was a while ago. Not that I think the stock has any value but wanted to post a picture of what they looked like for some color :)

It was an interesting place to work for at the time, I joined shortly after they launched their main product and it was pretty fast downhill from there. What can we say, we built the system to scale to the growth expectations that they had and that growth never materialized.

At the peak we had almost three racks of production equipment (15kW active 15kW backup power), in my mac daddy 47U 28" wide 49" deep Rittal TS-8 racks. Those racks are bad ass I gotta say. I could only fit three of them in a five rack cage (but due to power we had more than enough space in three). The cable management ROCKED (each server had at least 5 cables, which adds up fast).  The extra 4" of width made all the difference. The Servertech PDUs were great too, loved the integrated environmental sensors, rear door fans to exhaust the hot air away from the equipment. I love big racks! (couldn't resist)

The different facility was really nice as well the only downside is it doesn't support high density stuff.  I was hosted at the same place at the company before that and due to power constraints we had to have 8 feet of space in between our rows of racks. Not that I minded too much 8 feet was a lot of area to stretch out, put tables and shelves in for storage etc. With ~30 foot ceilings it's one of the few data centers I've been to that didn't make me feel claustrophobic.

My strategy for the company at the time was to make their main product as good as possible and then partner with someone like linkedin to leverage their network with Jobster's technology. Jobster's main product was never visible to the public it was a subscription only product, so most probably never saw the technology that the company was built on (initially at least). Their strategy was OOH, SHINY! rinse and repeat (at least 3 times while I was there).

Filed under: News Comments Off
22Oct/10Off

IPv4 address space exhaustion – tired

TechOps Guy: Nate

Just saw YASOSAIV6 (Yet another story on Slashdot about IPv6)..

They've been saying it for years, maybe even a decade? That we are running out of IPs and we need to move to IPv6. It's taken forever for various software and hardware manufacturers to get IPv6 into their stacks, and even now most of them haven't seen much real world testing. IPv6 is of course a chicken and egg problem.

My take on it, from a technological standpoint I do not look forward to IPv6, not at all. Really for one simple reason - IPv4 IP addresses are fairly simple to remember, and simpler to recognize. IPv6 - forget about it. I'm a simple minded person and that is a simple reason I don't look forward to IPv6.

I don't have a problem with Network Address Translation (NAT), it's amazing to me how many people out there absolutely despise NAT, I won't spend much time talking about why I think NAT is a good thing because I have better things to spend my time on :) [And yes when I'm not using NAT I absolutely run my firewalls in bridging mode again for simplicity purposes]

I don't believe we have an IPv4 crisis yet, sure IANA or whoever is the organization that assigns IP addresses says we are low on that free pool but guess what, service providers around the world have gobs of unused IPs. I talk to service providers fairly often and none of them are concerned about it, they do want you to be smart about IP allocation however. I suppose if your some big company and want to get 5,000 IP addresses you may need to be concerned, but for smaller organizations who may need a dozen or two dozen IPs at the most - really nothing to worry about.

One thing I think could free up a bunch of IPs and allow IPv4 to scale even further is to somehow fix the SSL/TLS/HTTPS protocol(s) so that it can support virtual hosts (short of using wild card certs). I'm sure it's possible but it won't be easy to get the software out to the field to all the various edge devices in order to be able to support it. One company I worked at needed about a hundred IPs JUST for SSL (wild card certs were not an option at the time due to lack of client side support).

I know we'll get to IPv6 eventually, and I'll accept that when we get there, though it may be far enough out that I don't deal with lower level stuff anymore so won't need to be concerned about it, I don't know.

Tagged as: , Comments Off
21Oct/10Off

Red Hat wants to end “IT Suckage”

TechOps Guy: Nate

Read an interesting article over on The Register with a lot of comments by a Red Hat executive.

And I can't help but disagree on a bunch of stuff the executive says. But it could be because the executive is looking at and talking with big bloated slow moving organizations that have a lot of incompetent people in their ranks ("Never got fired for buying X" mantra), instead of smaller more nimble more leading edge organizations willing, ready and able to take some additional "risk" for a much bigger return (such as running virtualized production systems, seems like a common concept to many but I know there's a bunch of people out there that aren't convinced that it will work, btw I ran my first VMware in production in 2004, and saved my company BIG BUCKS with the customer (that's a long story, and an even longer weekend)).

OK so this executive says

After all, processor and storage capacity keep tracking along on their respective Moore's and Kryder's Laws, doubling every 18 months, and Gilder's Law says that networking capacity should double every six months. Those efficiencies should lead to comparable economies. But they're not.

I was just thinking this morning about the price and capacity of the latest systems(sorry keep going back to the BL685c G7 with 48 cores and 512GB of ram :) ).

I remember back in 2004/2005 time frame the company I was at paying well over $100,000 for a 8-way Itanium system with 128GB of memory to run Oracle databases. The systems of today whether it is the aforementioned blade or countless others can run circles around such hardware now at a tiny fraction of the price. It wasn't unreasonable just a few short years ago to pay more than $1M for a system that had 512GB of memory and 24-48 CPUs, and now you can get it for less than $50,000(in this case using HP web pricing). That big $1M system probably consumed at least 5-10kW of power and a full rack as well, vs now the same capacity can go for ~800W(100% load off the top of my head) and you can get at least 32 of them in a rack(barring power/cooling constraints).

Granted that big $1M system was far more redundant and available than the small blade or rack mount server, but at the time if you wanted so many CPU cores and memory in a single system you really had no choice but to go big, really big. And if I was paying $1M for a system I'd want it to be highly redundant anyways!

With networking, well 10GbE has gotten to be dirt cheap, just think back a few years ago if you wanted a switch with 48 x 10GbE ports you'd be looking at I'd say $300k+ and it'd take the better part of a rack. Now you can get such switches in a 1U form factor from some vendors(2U from others), for sub $40k?

With storage, well spinning rust hasn't evolved all that much over the past decade for performance unfortunately but technologies like distributed RAID have managed to extract an enormous amount of untapped capacity out of the spindles that older architectures are simply unable to exploit. More recently the introduction of SSDs and the sub LUN automagic storage tiering technology that is emerging (I think it's still a few years away from being really useful) you can really get a lot more bang out of your system. EMC's fast cache looks very cool too from a conceptual perspective at least I've never used it and don't know anyone who has but I do wish 3PAR had it! Assuming I understand the technology right, with the key being the SSDs are used for both read and write caching. Verses something like the NetApp PAM card which is only a read cache. Neither Fast cache nor PAM is enough to make we want to use those platforms for my own stuff.

The exec goes on to say

Simply put, Whitehurst's answer to his own question is that IT vendors suck, and that the old model of delivering products to customers is fundamentally broken.

I would tend to agree for the most part but there are those out there that really are awesome. I was lucky enough to find one such vendor, and a few such manufacturers. As one vendor I deal with says they work with the customer not with the manufacturer, they work to give the customer what is best for them. So many vendors I have dealt with over the years are really lazy when it comes down to it, they only know a few select solutions from a few big name organizations and give blank stares if you go outside their realm of comfort (random thought: I got the image of Speed Bump: The roadkill possum from a really old TV series called Liquid Television that I watched on MTV for a brief time in the 90s).

By the same token while most IT vendors suck, most IT managers suck too, for the same reason. Probably because most people suck that may be what it comes down to it at the end of the day. IT as you well know is still an emerging industry, still a baby really evolving very quickly, but has a ways to go. So like with anything the people out there that can best leverage IT are few and far between. Most of the rest are clueless -- like my first CEO about 10-11 years ago was convinced he could replace me with a tech head from Fry's Electronics (despite my 3 managers telling him he could not). About a year after I left the company he did in fact hire such a person -- only problem was that individual never showed up for work (maybe he forgot).

Exec goes on to say..

"Functionality should be exploding and costs should be plummeting — and being a CIO, you should be a rock star and out on the golf course by 3 pm," quipped Whitehurst to his Interop audience.

That is in fact what is happening -- provided your choosing the right solutions, and have the right people to manage them, the possibilities are there, just most people don't realize it or don't have the capacity to evolve into what could be called the next generation of IT, they have been doing the same thing for so long, it's hard to change.

Speaking of being a rock star and out on the golf course by 3pm, I recall two things I've heard in the past year or so-

The first one used the golf course analogy, from a local VMware consulting shop that has a bunch of smart folks working for them I thought this was a really funny strategy and can see it working quite well in many cases - the person took an industry average of say 2-3 days to provision a new physical system, and said in the virtual world -- don't tell your customers that you can provision that new system in ten minutes, tell them it will take you 2-3 days, spend the ten minutes doing what you need and spend the rest of the time on the golf course.

The second one was from a 3PAR user I believe. Who told one of their internal customers/co-workers something along the lines of "You know how I tell you it takes me a day to provision your 10TB of storage? Well I lied, it only takes me about a minute".

For me, I'm really too honest I think, I tell people how long I think it will really take and at least on big projects am often too optimistic on time lines. Maybe I should take up Scotty's strategy and take my time lines and multiply them by four to look like a miracle worker when it gets done early. It might help to work with a project manager as well, I haven't had one for any IT projects in more than five years now. They know how to manage time (if you have a good one, especially one experienced with IT not just a generic PM).

Lastly the exec says

The key to unlocking the value of clouds is open standards for cloud interoperability, says Whitehurst, as well as standardization up and down the stack to simplify how applications are deployed. Red Hat's research calculates that about two-thirds of a programmer's time is spent worrying about how the program will be deployed rather than on the actual coding of the program.

Worrying about how the program will be deployed is a good thing, an absolutely good thing. Rewinding again to 2004 I remember a company meeting where one of the heads of the company stood up and said something along the lines of 2004 was the year of operations, we worked hard to improve how the product operates, and the next phase is going back to feature work for customers. I couldn't believe my ears, that year was the worst for operations, filled with half implemented software solutions that actually made things worse instead of better, outages increased, stress increased, turnover increased.

The only thing I could do from an operations perspective and buy a crap load of hardware and partition the application to make it easier to manage. We ended up with tons of excess capacity but the development teams were obviouslly unable to make the design changes we needed to improve the operations of the application, but we at least had something that was more manageable, the deployment and troubleshooting teams were so happy when the new stuff was put into production, no longer did they have to try to parse gigabyte sized log files trying to find which errors belong to which transactions from which subsystem. Traffic for different subsystems was routed to different physical systems so you knew if there was an issue with one type of process you go to server farm X to look at it, problem resolution was significantly faster.

I remember having one conversation with a software architect in early 2005 about a particular subsystem that was very poorly implemented (or maybe even designed), it caused us massive headaches in operations, non stop problems really. His response was Well I invited you to a architecture meeting in January of 2004 to talk about this but you never showed up. I don't remember the invite but if I saw it I know why I didn't show up, it's because I was buried in production outages 24/7 and had no time to think more than 24 hours ahead yet alone think about a software feature that was months away from deployment. Just didn't have the capacity, was running on fumes for more than a year.

So yes, if you are a developer please do worry about how it is deployed, never stop worrying. Consult your operations team (assuming they are worth anything), and hopefully you can get a solid solution out the door. If you have a good experienced operations team then it's very likely they know a lot more about running production than you do and can provide some good insight into what would provide the best performance and uptime from an operations perspective. They may be simple changes, or not.

One such example, I was working at a now defunct company who had a hard on for Ruby on Rails. They were developing app after app on this shiny new platform. They were seemingly trying to follow Services Oriented Architecture (SOA), something I learned about ironically at a Red Hat conference a few years ago (didn't know there was a acronym for that sort of thing it seemed so obvious). I had a couple, really simple suggestions for them to take into account for how we would deploy these new apps. Their original intentions called for basically everything running under a single apache instance(across multiple systems), and for example if Service A wanted to talk to Service B then it would talk to that service on the same server. My suggestions which we went with involved two simple concepts:

  • Each application had it's own apache instance, listening on it's own port
  • Each application lived behind a load balancer virtual IP with associated health checking, with all application-to-application communication flowing through the load balancer

Towards the end we had upwards of I'd say 15 of these apps running on a small collection of servers.

The benefits are pretty obvious, but the developers weren't versed in operations -- which is totally fine they don't need to be (though it can be great when they are, I've worked with a few such people though they are VERY RARE) that's what operations people do and you should involve them in your development process.

As for cloud standards -- folks are busy building those as we speak and type. VMware seems to be the furthest along from an infrastructure cloud perspective I believe, I wouldn't expect them to lose their leadership position anytime soon they have an enormous amount of momentum behind them, and it takes a lot to counter that momentum.

About a year ago I was talking to some former co-workers who told me another funny story they were launching a new version of software to production, the software had been crashing their test environments daily for about a month. They had a go no-go meeting in which everyone involved with the product said NO GO. But management overrode them, and they deployed it anyways. The result? A roughly 14 hour production outage while they tried to roll the software back. I laughed and said, things really haven't changed since I left have they?

So the solutions are there, the software companies and hardware companies have been evolving their stuff for years, the problem is the concepts can become fairly complex when talking about things like capacity utilization and stranded resources, getting the right people in place to be able to not only find such solutions but deploy and manage them as well can really go a long ways, but those people are rare at this point.

I haven't been writing too much recently been really busy, Scott looks to be doing a good job so far though.

 

Tagged as: , Comments Off
11Oct/10Off

Qlogic answers my call for help

TechOps Guy: Nate

THANK YOU QLOGIC. I have been a long time user of Qlogic stuff and like them a lot. If you have been reading this blog for a while you may of noticed earlier in the year I was criticizing the network switch industry (includes my favorite manufacturers as well) for going down the route of trying to "reclaim the network" by working on standards that would move the inter-VM switching traffic out of the host and back into the network switches. I really think the whole concept is really stupid, and a desperate attempt to hold onto what will be a dramatically declining ports market in the coming years. Look no further than my recent post on testing the limits of virtualization.

My answer to the dilemma ? Put a layer 2 hardware switching fabric into the server, less latency, faster performance.

And Qlogic has done just that. I will refrain from using colorful metaphors to describe my glee, but I certainly hope this is a trend going forward.

According to our friends at The Register, Qlogic has released new Converged Network Adapters (CNA) that includes an integrated layer 2 switch for virtual machines.

EMEA Marketing head for QLogic, Henrik Hansen, said: "Within the ASIC we have embedded a layer 2 Ethernet switch [and] can carve up the two physical ports into 4 NIC partitions or NPARs, which can each be assigned to a specific VM. There can be eight of them with dual-port product."An Ethernet message from one VM to another in the same server goes to the QLogic ASIC and is switched back to the target VM. This is reminiscent of Emulex' VNIC feature.

From the specs:

  • PCI Express Gen2 x8
  • Dual 10Gbps and quad 1Gbps ports on a single controller
  • Integrated 10GBase-KR and 10GBase-T PHYs
  • Concurrent TCP/IP, FCoE, and iSCSI protocol support with full hardware offload
  • Industry standard SR-IOV and QLogic's switch-agnostic NIC Partitioning (NPAR)
  • Wake-on-LAN including Magic Packet recognition
  • Common drivers and API’s with existing QLogic NIC, FCoE, and iSCSI products

Side note: I love that they have 10GbaseT too!!

I think the ASIC functionality needs more work as it seems limited to supporting only a couple VMs rather than being a more generic switching fabric but we gotta start somewhere!

The higher end 8200 CNA looks like it has much of the same technology available in the HP FlexFabric (which I know at least part of is already based on Qlogic technology though might not be these specific ASICs I don't know)

VMflex. With QLogic’s new VMflex technology, one Converged Network Adapter is viewed by the server operating system (OS) as a flexible mix (up to  four per physical port) of standalone NICs, FCoE adapters, and iSCSI adapters, with the ability to allocate guaranteed bandwidth to each virtual adapter.  This unique feature can be switch dependent or switch agnostic— it is not necessary to pair an 8200 Series adapter with any specific 10GbE switch model to enable partitioning.

I would love to see more technical information on the VMFlex and the layer 2 switching fabric, I tried poking around on Qlogic's site but didn't come up with anything too useful.

So I say again, thank you Qlogic, and I hope you have started a trend here. I firmly believe that offloading the switching functionality to an ASIC rather than performing it in software is critical, and when you have several hundred VMs running on a single server not wasting your uplink bandwidth to talk between them is just as critical. The functionality of the ASIC need not offer too much, for me I think the main things would be vlan tagging and sFlow, some folks may want QoS as well.

My other request, I don't know if it is already possible or not is to be able to run a mix of jumbo frames and standard frame sizes on different virtual NICs riding on the same physical network adapter, without configuring everything for jumbo frames, because that causes compatibility issues (especially for anything using UDP!).

The networking industry has it backwards in my opinion, but I can certainly understand the problem they face.

Tagged as: , Comments Off
10Oct/10Off

Intel or ASIC

TechOps Guy: Nate

Just another one of my random thoughts I have been having recently.

Chuck wrote a blog not too long ago how he believes everyone is going to go to Intel (or x86 at least) processors in their systems and move away from ASICs.

He illustrated his point by saying some recent SPEC NFS results showed the Intel based system outperforming everything else. The results were impressive, the only flaw in them is that the costs are not disclosed for SPEC. An EMC VMAX with 96 EFDs isn't cheap. And the better your disk subsystem is the faster your front end can be.

Back when Exanet was still around they showed me some results from one of their customers testing SPEC SFS on the Exanet LSI (IBM OEM'd) back end storage vs 3PAR storage, and for the same number of disks the SPEC SFS results were twice as high on 3PAR.

But that's not my point here or question. A couple of years ago NetApp posted some pretty dismal results for the CX-3 with snapshots enabled. EMC doesn't do SPC-1 so NetApp did it for them. Interesting.

After writing up that Pillar article where I illustrated the massive efficiency gains on the 3PAR architecture(which is in part driven by their own custom ASICs), it got me thinking again, because as far as I can tell Pillar uses x86 CPUs.

Pillar offers multiple series of storage controllers to best meet the needs of your business and applications. The Axiom 600 Series 1 has dual-core processors and supports up to 24GB cache. The Axiom 600 Slammer Series 2 has quad-core processors and double the cache providing an increase in IOPS and throughput over the Slammer Series 1.

Now I can only assume they are using x86 processors, for all I know I suppose they could be using Power, or SPARC, but I doubt they are using ARM :)

Anyways back to the 3PAR architecture and their micro RAID design. I have written in the past about how you can have tens to hundreds of thousands of mini RAID arrays on a 3PAR system depending on the amount of space that you have. This is, of course to maximize distribution of data over all resources to maximize performance and predictability. When running RAID 5 or RAID 6, there are of course parity calculations involved. I can't help but wonder what sort of chances in hell a bunch of x86 CPU cores have in calculating RAID in real time for 100,000+ RAID arrays, with 3 and 4TB drives not too far out, you can take that 100,000+ and make it 500,000.

Taking the 3PAR F400 SPC-1 results as an example, here is my estimate on the number of RAID arrays on the system, fortunately it's mirrored so math is easier:

  • Usable capacity = 27,053 GB (27,702,272 MB)
  • Chunklet size = 256MB
  • Total Number of RAID-1 arrays = ~ 108,212
  • Total Number of data chunklets = ~216,424
  • Number of data chunklets per disk = ~563
  • Total data size per disk = ~144,128 MB (140.75 GB)

For legacy RAID designs it's probably not a big deal, but as disk drives grow ever bigger I have no doubt that everyone will have to move to a distributed RAID architecture, to reduce disk rebuild times and lower the chances of a double/triple disk failure wiping out your data. It is unfortunate (for them) that Hitachi could not pull that off in their latest system.

3PAR does use Intel CPUs in their systems as well, though they aren't used too heavily, on the systems I have had even at peak spindle load I never really saw CPU usage above 10%.

I think ASICs are here to stay for some time, on the low end you will be able to get by with generic CPU stuff, but on the higher end it will be worth the investment to do it in silicon.

Another place to look for generic CPUs vs ASICs is in the networking space. Network devices are still heavily dominated by ASICs because generic CPUs just can't keep up. Now of course generic CPUs are used for what I guess could be called "control data", but the heavy lifting is done by silicon. ASICs often draw a fraction of the power that generic CPUs do.

Yet another place to look for generic CPUs vs ASICs is in the HPC space - the rise of GPU-assisted HPC allowing them to scale to what was (to me anyways) unimaginable heights.

Generic CPUs are of course great to have and they have come a long way, but there is a lot of cruft in them, so it all depends on what your trying to accomplish.

The fastest NAS in the world is still BlueArc, which is powered by FPGAs, though their early cost structures put them out of reach for most folks, their new mid range looks nice, my only long time complaint about them has been their back end storage - either LSI or HDS, take it or leave it. So I leave it.

The only SPEC SFS results posted by BlueArc are for the mid range, nothing for their high end (which they tested on the last version of SFS, nothing yet for the current version).

 

9Oct/10Off

Capacity Utilization: Storage

TechOps Guy: Nate

So I was browsing through that little drop down address bar in Firefox hitting the sites I usually hit, and I decided hey let's go look at what Pillar is doing. I've never used their stuff but I dig technology you know, so I like to try to keep tabs on companies and products that I haven't used, and may never consider using, good to see what the competition is up to, because you never know they may come out with something good.

Tired of the thin argument

So the CEO of Pillar has a blog, and he went on a mini rant about how 3PAR^H^H^H^HHP is going around telling people you can get 100TB of capacity in one of their 70TB arrays. I haven't read too deep into what the actual claim they are making is, but being so absolutely well versed in 3P..HP technology I can comment with confidence in what their strategy is and how they can achieve those results. Whether or not they are effective at communicating that is another story, I don't know because well I don't read everything they say.

Pillar notes that HP is saying that due to the 3PAR technologies you can get by with less and he's tired of hearing that old story over and over.

Forget about thin for the moment!

So let me spray paint another angle for everyone to see. As you know I do follow SPC-1 numbers pretty carefully. Again not that I really use them to make decisions, I just find the numbers and disclosure behind them very interesting and entertaining at times. It is "fun" to see what others can do with their stuff in a way that can be compared on a level playing field.

I wrote, what I consider a good article on SPC-1 benchmarks a while back, EMC gave me some flak because they don't believe SPC-1 is a valid test, when I believe EMC just doesn't like the disclosure requirements, but I'm sure you won't ever hear EMC say that.

SPC-1 Results

So let's take the one and only number that Pillar published, because, well that's all I have to go on, I have no personal experience with their stuff, and don't know anyone that uses it. So if this information is wrong it's wrong because the results they submitted were wrong.

So, the Pillar Axiom 600's results have not stood the test of time well at all, as you would of noticed in my original article, but to highlight:

  • System tested: January 12, 2009
  • SPC-1 IOPS performance: 64,992
  • SPC-1 Usable space: 10TB
  • Disks used: 288 x 146G 15k RAID 1
  • IOPS per disk: 226 IOPS/disk
  • Average Latency at peak load: 20.92ms
  • Capacity Utilization (my own metric I just thought of): 34.72 GB/disk
  • Cost per usable TB (my own metric extrapolated from SPC-1): $57,097 per TB
  • Cost per IOP (SPC-1 publishes this): $8.79

The 3PAR F400 by contrast was tested just 105 days later and absolutely destroyed the Pillar numbers, and unlike the Pillar numbers the F400 has held up very well against the test of time all the way to present day even:

  • System tested: April 27, 2009
  • SPC-1 IOPS performance: 93,050
  • SPC-1 Usable space: 27 TB
  • Disks used: 384 x 146G 15k RAID 1
  • IOPS per disk: 242 IOPS/disk
  • Average Latency at peak load: 8.85ms
  • Capacity Utilization: 70.432 GB/disk
  • Cost per usable TB: $20,312 per TB
  • Cost per IOP: $5.89

Controller Capacity

Now in my original post I indicated stark differences in some configurations that tested substantially less physical disks than the controllers supported, there are a couple of possibilities I can think of for this:

  • The people running the test didn't have enough disks to test (less likely)
  • The controllers on the system couldn't scale beyond the configuration tested, so to illustrate the best bang for your $ they tested with the optimal number of spindles to maximize performance (more likely)

So in Pillar's case I think the latter is the case as they tested with a pretty small fraction of what their system is advertised as being capable of supporting.

Efficiency

So taking that into account, the 3PAR gives you 27TB of usable capacity, note here we aren't even taking into account the thin technologies, just throw those out the window for a moment, let's simplify this.

The Pillar system gives you 10TB of usable capacity, the 3PAR system gives you about 270% more space and 130% more performance for less money.

What would a Pillar system look like(or Systems I guess I should say since we need more than one) that could give us 27TB usable capacity and 93,000 SPC-1 IOPS using 146G 15k RPM disks (again trying to keep level playing field here)?

Well I can only really guess, to reach the same level of performance Pillar would need an extra 124 disks, so 412 spindles. Maintaining the same level of short stroking that they are doing(34.7GB/disk), those extra 124 spindles only get you to roughly 14.3TB.

And I'm assuming here because my comments earlier about optimal number of disks to achieve performance, if you wanted to get those extra 124 spindles in you need a 2nd Axiom 600, and all the costs with the extra controllers and stuff. Controllers obviously carry a hefty premium over the disk drives. While the costs are published in Pillar's results I don't want to spend the time to try to extrapolate that angle.

And if you do in fact need more controllers, the system was tested with two controllers, if you have to go to four (tested 3PAR F400 has four), 3PAR has another advantage completely unrelated to SPC-1, the ability to maintain performance levels under degraded conditions (controller failure, software upgrade, whatever) with Persistent Cache. Run your same SPC-1 test, and yank a controller out from each system (3PAR and Pillar) and see what the results are. The numbers would be even more embarrassingly in 3PAR's favor thanks to their architecture and this key caching feature. Unlike most of 3PAR's feature add-ons, this one comes at no cost to the customer, the only requirement is you must have at least 4 controllers on the box.

So you still need to get to 27 TB of usable capacity. From here it can get really fuzzy because  you need to add enough spindles to get that high but then you need to adjust the level of short stroking your doing to use more of the space per drive, it wouldn't surprise me if this wasn't even possible on the Pillar system(not sure if any system can do it really,  but I don't know).

If Pillar can't adjust the size of the short stroking then the numbers are easy, at 34.7GB/disk they need 778 drives to get to 27TB of usable capacity, roughly double what 3PAR has.

Of course the performance of a two-system based Axiom 600 with 778 drives will likely outperform a 384-disk F400(I should hope so at least), but you see where I'm going.

I'm sure Pillar engineers could come up with a way to configure the system more optimally my 778 drive solution is crazy but from a math perspective it's the easiest and quickest thing I could come up with, with the data I have available to me.

This is also a good illustration why when I go looking at what Xiotech posts, I really can't compare them against 3PAR or anybody else, because they only submit results for ~16 drive systems. To me, it is not valid to compare a ~16 drive system to something that has hundreds of drives and try to extrapolate results. Xiotech really does give stellar results as far as capacity utilization and IOPS/disk and stuff, but they haven't yet demonstrated that those numbers are scalable beyond a single ISE enclosure - yet alone to several hundred disks.

I also believe the 3PAR T800 results could be better too, the person at 3PAR who was responsible for running the test was new to the company at the time and the way he laid out the system was, odd to say the least. The commands he used were even depreciated. But 3PAR isn't all that interested in re-testing, they're still the record holder for spinning rust in a single system(more than two years running now no doubt!).

Better response times to boot

You can see the 3PAR system performs with less than half the amount of latency that the Pillar system does despite the Pillar system short stroking their disks. Distributed RAID with full mesh architecture at work baby. I didn't even mention it but the Pillar system has double the cache than than the F400. I mean the comparison really almost isn't fair.

I'm sure Pillar has bigger and better things out now since they released the SPC-1 numbers for the Axiom, so this post has the obvious caveat that I am reporting based on what is published. They'd need to pull more than a rabbit out of a hat to make up these massive gaps though I think.

Another Angle

We could look at this another way as well, assuming for simplicity's sake for a moment that both systems scale lineally up or down, we can configure a 3PAR F400 with the same performance specs as the Pillar that was tested.

You'd need 268 disks on the 3PAR F400 to match the performance of the Pillar system. With those 268 disks you'd get 18.4 TB of usable space, same performance, fewer disks, and 184% additional usable capacity. And scaling the cost down like we scaled the performance down, the cost would drop to roughly $374,000, a full $200,000 less than Pillar for the same performance and more space.

Conclusion

So hopefully this answers the question with more clarity why you can get less storage from the 3PAR F400 and get the same or better performance and usable capacity than going with a Pillar Axiom 600.  At the end of the day 3PAR drives higher capacity utilization and delivers superior results for significantly less greenbacks. And I didn't even take 3PAR's thin technologies into account, the math there can become even more fuzzy depending on the customer's actual requirements and how well they can leverage thin built in.

You may be able to understand why HP was willing to go to the end of the earth to acquire 3PAR technology. And you may be able to understand why I am so drawn to that very same technology. And here I'm just talking about performance. Something that unlike other things(ease of use etc) is really easy to put hard numbers on.

The numbers are pretty simple to understand, and you can see why the big cheese at HP responsible for spear heading the 3PAR purchase said:

The thin provisioning, automatic storage tiering, multi-tenancy, shared-memory architecture, and built-in workload management and load balancing in the 3PAR arrays are years ahead of the competition, according to Donatelli, and therefore justify the $2.4bn the company paid to acquire 3PAR in a bidding war with rival Dell.

Maybe if I'm lucky I can trigger interest in The Register again by starting a blog war or something and make tech news! woohoo! that would be cool. Of course now that I said that it probably won't happen.

I'm told by people who know the Pillar CEO he is "raw", much like me. so it will be interesting to see the response. I think the best thing they can do is post new SPC-1 numbers with whatever the latest technology they have is, preferably on 146G 15k disks!

Side note

It was in fact my 3PAR rep that inspired me to write about this SPC-1 stuff, I was having a conversation with him earlier in the year where he didn't think the F400 was as competitive against the HDS AMS2500 as he felt it needed to be. I pointed out to him that despite the AMS2500 having similar SPC-1 IOPS and similar cost, the F400 offered almost twice the usable capacity. And the cost per usable TB was far higher on the 2500. He didn't realize this.  I did see this angle so felt the need to illustrate it. Hence my Cost per SPC-1 Usable TB. It's not a performance metric, but in my opinion from a cost perspective a very essential metric, at least for highly efficient systems.

(In case it wasn't obvious, I am by no means compensated by 3PAR in any way for anything I write, I have a deep passion for technology and they have some really amazing technology, and they make it easy to use and cost effective to boot)

8Oct/10Off

I/O Virtualization for mortals

TechOps Guy: Nate

This product isn't that new but haven't seen many people talk about it, I first came across it a few weeks ago certainly looked very innovative.

I don't know who started doing I/O virtualization first, maybe it was someone like Xsigo, or maybe it was HP with their VirtualConnect or maybe it was someone else, but the space has heated up in the past couple of years.

Neterion, a name that sounds familiar but I can't quite place it... a company by the name of Exar may of bought them or something. But anyways there is this interesting virtualized NIC that they have - the X3120 V-NIC looks pretty cool -

Neterion's family of 10 Gigabit Ethernet adapters offer a unique multi-channel device model. Depending upon the product, a total of between eight and seventeen fully independent, hardware-based transmit and receive paths are available; each path may be prioritized for true Quality-of-Service support.

I/O Virtualization Support

  • Special "multi-function PCI device" mode brings true IOV to any industry-standard server. In multi-function mode, up to 8 physical functions are available (more in ARI-capable systems). Each physical function appears to the system as an independent Ethernet card
  • Unique, hardware-based multi-channel architecture mitigates head-of-line blocking and allows direct data transfer between hardware channels and host-based Virtual Machines without hypervisor intervention (greatly reducing CPU workload)
  • VMware® NetQueue support
  • Dedicated per-VF statistics and interrupts
  • Support for function-level reset (FLR)
  • Fully integrated Layer 2 switching function

I removed some bullet points of things to shorten the entry a bit and those things I wasn't exactly sure what they did anyways! Anyone know what ARI means above?

Never used the product, but it is very nice to see such a product in the market place, to get Virtual Connect "like" functionality (at least as far as the virtual NICs go, I know theres a lot of other advantages to VC) in your regular rack mount systems from any vendor and at least potentially connect to any 10GbE switch, as far as I can tell there's no special requirements for a specific type of switch.

8Oct/10Off

How inefficient can you get?

TechOps Guy: Nate

[ the page says the system was tested in Jan 2010, so not recent, but I don't recall seeing it on the site before now, in any case it's still crazy]

I was about to put my laptop down when I decided hey let's go over to SPEC and see if there are any new NFS results posted.

So I did, you know me I am into that sort of stuff. I'm not a fan of NFS but for some reason the SPEC results still interest me.

So I go and see that NEC has posted some results. NEC isn't a very well known server or even IT supplier in the U.S. at least as far as I know. I'm sure they got decent market share over in Asia or something.

But anyways they posted some results, and I have to say I'm shocked. Either there is a glaring typo or that is just the worst NAS setup on the face of the planet.

It all comes down to usable capacity. I don't know how you can pull this off but they did - they apparently have 284 300GB disks on the system but only have 6.1 TB of usable space! That is roughly 83TB of raw storage and they only manage to get something like 6% capacity utilization out of the thing?

Why even bother with disks at all if your going to do that? Just go with a few SSDs.

But WAIT! .. WAIT! It gets better. That 6.1 TB of space is spread across -- wait for it -- 24 file systems.

12 filesystems were created and used per node. One of 24 filesystems consisted of 8 disks which were divided into two 4-disk RAID 1+0 pools, and each of the other 23 filesystems consisted of 12 disks which were divided into two 6-disk RAID 1+0 pools. There were 6 Disk Array Controllers. One Disk Array Controller controlled 47 disks, and each one of the other 5 controlled 48 disks.

I mean the only thing I can hope for is that the usable capacity is in fact a big typo.

Total Exported Capacity 6226.5GB

But if it's not I have to hand it to them for being brave enough to post such terrible results. That really takes some guts.

 

8Oct/10Off

Windows mobile: the little OS that couldn’t

TechOps Guy: Nate

I don't know how to feel, a mixture of happiness and sorrow I suppose. Happy that MS is still unable to get any traction in the mobile space, and sad that they have spent so much time and effort and have gotten nowhere, I feel sorry for them, just a little sorry though.

From our best friends at The Register, an article quoting some folks at Gartner -

Gartner said that Windows Phone 7 will provide a fillip to Microsoft's worldwide mobile market share, pushing it up from 4.7 per cent this year to 5.2 per cent in 2011, but it's share will fall again to 3.9 per cent by 2014.

That's just sad, for the worlds biggest software company. I mean I have a Palm pre and I know their market share is in the toilet, not that it really matters at this point since they sold out to HP they made their money. But Palm had a microscopic amount of resources compared to Microsoft. Microsoft has been trying this for I have to think at least a decade now. If the above is true, if by 2014 they have 4% market share, what would you do if you were MS, spending 14 years to get 4% market share?

I never understood the mobile space, and I worked at one of the earliest companies that capitalized on the space earlier this decade, selling ringtones and wallpapers like nobody's business. All I could think was what are these people thinking buying all this crap. But they just kept coming and coming. Like moths to a flame or something. Nobody was worse than Sprint though before they ended up partnering with that company I was with. I remember back in ... 2004? I looked to see what Sprint had to offer as far as ringtones and stuff and they actually wanted you to rent them, that's right, pay $2.99 or whatever for a ringtone and then have to buy it again in 90 days. That practice stopped after they bought Nextel which was already a customer of ours at the time and Sprint was merged into the service that we provided.

If it wasn't for Classic, I would still be using a trusty Sanyo feature phone, booted up in about 15 seconds, crashed maybe once or twice a year, worked really well as a phone, and battery lasted a good long time too.

I noticed the battery life on my Palm went through the roof practically after I stopped using the MS Exchange plugin all this time I thought the battery life was crap when it was that stupid plugin draining it.

Looking forward to the PalmPad from HP myself. I won't go near an Apple product. I don't trust Google either, so Android is out :)

 

Filed under: General Comments Off