TechOpsGuys.com Diggin' technology every day

The thoughts around Dell buying Compellent made me think back to Dell’s acquistiion of the IP and some engineering employees of Exanet, as The Register put it, a crashed NAS company.

I was a customer and user of Exanet gear for more than a year, and at least in my experience it was a solid product, very easy to use, decent performance and scalable. The back end architecture to some extent mirrored the 3PAR hardware-based architecture but in software, really a good design in my opinion.

Basic Exanet Architecture

Their standard server at the time they went under was a IBM x3650, dual proc quad core Intel Xeon 5500-based platform with 24GB of memory.

Each server had multiple software processes called fsds or File system daemons, that ran, they ran one fsd per core. Each fsd was responsible for a portion of the file system (x number of files), they load balanced it quite well I never had to manually re-balance or anything. Each fsd was allocated its own memory space used for itself as well as cache, if I recall right the default was around 1.6GB per fsd.

Each NAS head unit had back end connectivity to all of the other NAS units in the cluster(minimum 2, maximum tested at the time they went under was 16). A request for a file could come in on any node, any link. If the file wasn’t home to that node it would transparently forward the request to the right node/fsd to service the request on the back end. Much like how 3PAR’s backplane forwards requests between controllers.

Standard for back end network was 10Gbps on their last models.

As far as data protection, the use of “commodity” servers did have one downside, they had to use UPS systems as their battery backup to ensure enough time for the nodes to shut down cleanly in the event of a power failure. This could present problems at some data centers as operating a UPS in your own rack can be complicated from a co-location point of view(think EPO etc). Another similar design that Exanet had compared to 3PAR is their use of internal disks to flush cache to, which is something I suppose Exanet was forced into doing, other storage manufacturers use battery backed cache in order to survive power outages of some duration. But both Exanet and 3PAR dump their cache to an internal disk so that the power outage can last for a day, a week, or even a month and it won’t matter, data itnegrity is not compromised.

32-bit platform

The only thing that held it back was they didn’t have enough time or resources to make the system fully 64-bit before they went under, that would of unlocked a whole lot of additional performance they could of gotten. Being locked into a 32-bit OS really limited what they could do on a single node, and as processors became ever more powerful they really had to make the jump to 64-bit.

Exanet was entirely based on “commodity” hardware, not only were they using x86 CPUs but their NAS controllers were IBM 2U rackmount servers running CentOS 4.4 or 4.5 if I recall right.

To me, as previous posts have implied, if your going to base your stuff on x86 CPUs, go all out, it’s cheap anyways. I would of loved to have seen a 32-48 core Exanet NAS controller with 512GB-1TB of memory on it.

Back to Dell

Dell originally went into talks with Exanet a while back because Exanet was willing to certify Equallogic storage as a back end provider of disk to an Exanet cluster, using iSCSI inbetween the Exanet cluster and the Equallogic storage. Since nobody else in the indusry seemed willing to have their NAS solution talk to a back end iSCSI system. As far as I know the basic qualifications for this solution was completed in 2009, quite a ways before they ran out of cash.

Why did Exanet go under? I believe primarily because the market they were playing in was too small with too few players in it, not enough deals to go around, so whomever had the most resources to outlast the rest would come out on top, in this case I believe it was Isilon, even though they too were taken out by EMC from the looks of their growth it didn’t seem like they were in a fine position to continue to operate independently. With Ibrix and Polyserve going to HP, Onstor going to LSI, and I’m still convinced BlueArc will go to HDS at some point(they are once again filing for IPO but word on the street is they aren’t in very good shape), I suspect after they fail to IPO and go under. They have a very nice NAS platform, but HDS has their hands tied in supporting 3rd party storage other than HDS product, BlueArc OEM’s LSI storage like so many others.

About a year ago SGI OEM’d one of BlueArc’s products though recently I have looked around the SGI site and see no mention of it. Either they have abandoned it (more likely) or are just really quiet. Since I know SGI is also a big LSI shop I wonder if they are making the switch to Onstor. One industry insider I know suspects LSI is working on integrating the Onstor technology directly into their storage systems rather than having an independent head unit, which makes sense if they can make it work.

But really my question is why hasn’t Dell announced anything related to the Exanet technology? They could of, quite possibly within a week or two had a system running and certified on Dell PowerEdge equipment and selling to both existing Exanet customers as well as new ones. The technology worked fine, it was really easy to setup and use, and it’s not as if Dell has another solution in house that competes with it. AND since it was an entirely software based solution there was really no costs involved in manufacturing. Exanet had more than one PB-sized deal in the works at the time they went under, that’s a lot of good will Dell just threw away. But hey, what do you expect, it’s Dell. Thankfully they didn’t get their dirty paws on 3PAR.

When I looked at how a NetApp system was managed compared to the Exanet my only response was You’re kidding, right?

Time will tell if anything ever comes of the technology.

I really wanted 3PAR to buy them of course, they were very close partners with 3PAR and both pitched each other’s products at every opportunitiy. Exanet would go out of their way to push 3PAR storage whenever possible because they knew how much trouble the LSI storage could be, and they were happy to get double the performance per spindle off 3PAR vs LSI. But I never did get an adequate answer out of 3PAR as to why they did not pursue Exanet, they were in the early running but pulled out for whatever reason, the price tag of less then $15M was a steal.

Now that 3PAR is with HP we’ll see what they can do with Ibrix, I knew of more than one customer that migrated off of things like Ibrix and Onstor to Exanet, HP has been pretty silent about Ibrix since they bought them as far as I know. I have no idea how much R&D they have pumped into it over the years or what their plans might be.