TechOpsGuys.com Diggin' technology every day

11Jun/12Off

3PAR and NPIV

TechOps Guy: Nate

I was invited to a little preview of some of the storage things being announced at HP Discover last week, just couldn't talk about it until the announcement. Since I was busy in Amsterdam all last week I really didn't have a lot of time to think about blogging here.

But I'm back and am mostly adjusted to the time zone differences I hope. HP had at least two storage related announcements they made last Monday, one related to scaling of their StoreOnce dedupe setup and another related to 3PAR. The StoreOnce announcement seemed to be controversial, since I really have a minimal amount of exposure to that sort of product I won't talk about it much, on the surface it sounded pretty impressive but if the EMC claims are true than it's unfortunate.

Anyways onto the 3PAR announcement which while it had a ton of marketing around it, it basically comes down to three words:

3PAR Supports NPIV (finally)

NPIV in a nutshell the way I understand it is a way of virtualizing connections between points in a fibre channel network, most often in the past it seems to have been used to present storage directly to VM hosts, via FC switches. NPIV is also used by HP's VirtualConnect technology on the FC side to connect the VC modules to a NPIV-aware FC switch (which is pretty much all of them these days?), and then the switch connected to the storage(duh). I assume that NPIV is required by Virtual Connect because the VC module isn't really a switch it's more of a funky bridge.

Because 3PAR did not support NPIV (for what reason I don't know I kept asking them about it for years but never got a solid response as to why not or when they might support it) there was no way to directly connect a Virtual Connect module (either the new Flex Fabric or the older dedicated FC VC modules) to a 3PAR array, you had to have a switch as a middleman. Which just seemed like a waste. I mean here you have a T or now a V-class system with tons of ports, you have these big blade chassis with a bunch of servers in them, with the VC modules acting like a switch (acting as in aggregating points) and you can't directly connect it to the 3PAR storage! It was an unfortunate situation. Even going back to the 3cV, which was a bundle of sorts of 3PAR, HP c-Class Blades and VMware (long before HP bought 3PAR of course), I would have thought getting NPIV support would of been a priority but it didn't happen, until now (well last Monday I suppose).

So at scale you have up to 96 host fibre channel ports on a V400 or 192 FC ports on a V800 operating at 8Gbps. At a maximum you could get by with 48 blade enclosures (2 FC/VC modules each with a single connection) on a V400 or of course double that to 96 on a V800. Cut it in half if you want higher redundancy with dual paths on each FC/VC module. That's one hell of a lot of systems directly connected to the array. Users may wish to stick to a single connection per VC module allowing the 2nd connection to be connected to something else, maybe another 3PAR array. You still have full redundancy with two modules and one path per module. 3PAR 4Gbps HBAs (note the V-class has 8Gbps) have queue depths of something like 1,536 (not sure what the 8Gbps HBAs have). If your leveraging full height blades you get 8 per chassis, absolute worst case scenario you could set a queue depth of 192/server (I use 128/server on my gear). You could probably pretty safely go quite a bit higher though more thought may have to be had in certain circumstances. I've found 128 has been more than enough for my own needs.

It's cost effective today to easily get 4TB worth of memory per blade chassis, memory being the primary driver of VM density, so your talking anywhere from 96 - 384 TB of memory hooked up to a single 3PAR array. From a CPU perspective anywhere from 7,680 CPU cores all the way up to 36,684 CPU cores in front of a single storage system, a system that has been tested to run at over 450,000 SPC-1 IOPS. The numbers are just insane.

All we need now is a flat ethernet fabric to connect the Virtual Connect switches to, oh wait we have that too, though it's not from HP. A single pair of Black Diamond X-Series switches could scale to the max here as well, supporting a full eight 10Gbit/second connections per blade chassis with 96 blade chassis directly connected - which, guess what - is the maximum number of 10GbE ports on a pair of FlexFabric Virtual Connect modules (assuming your using two ports for FC). Of course all of the bandwidth is non blocking. I don't know what the state of interoperability is but Extreme touts their VEPA support in scaling up to 128,000 VMs in an X-series, and Virtual Connect appears to tout their own VEPA support as well. Given the lack of more traditional switching functionality in the VC modules it would probably be advantageous to leverage VEPA (whether or not this extends to the Hypervisor I don't know - I suspect not based on what I last heard at least from VMware, I believe it is doable in KVM though) to route that inter-server traffic through the upstream switches in order to gain more insight into it and even control it. If you have upwards of 80Gbps of connectivity per chassis anyways it seems there'd be abundant bandwidth to do it. All HP needs to do now is follow the Dell and revise their VC modules to natively support 40GbE (the Dell product is a regular Blade Ethernet switch by contrast and is not yet shipping).

You'd have to cut at least one chassis out of that configuration(or reduce port counts) in order to have enough ports on the X-Series to uplink to other infrastructure. (When I did the original calculations I forgot there would be two switches not one, so there's more than enough ports to support 96 blade chassis between a pair of X-8s going full bore with 8x10GbE/chassis and you could even use M-LAG to go active-active. if you prefer). I'm thinking load balancers, and some sort of scale-out NAS for file sharing, maybe the interwebs too.

Think about that, up to 30,000 cores, more than 300 TB of memory, sure you do have a bunch of bridges, but all of it connected by only two switches, and one storage array (perhaps two). Just insane.

One HP spokesperson mentioned that even a single V800 isn't spec'd to support their maximum blade system configuration of 25,000 VMs. 25k VMs on a single array does seem quite high(that comes to an average of 18 SPC-1 IOPS/VM), but it really depends on what those VMs are doing. I don't see how folks can go around tossing solutions about saying X number of VMs when workloads and applications can vary so widely.

So in short, the announcement was simple - 3PAR supports NPIV now - the benefits of that simple feature addition are pretty big though.

11Jun/12Off

Back from Amsterdam

TechOps Guy: Nate

I'm back from Amsterdam - it was about what I expected. I basically stuck to the hotel and the data center - I even skipped out on that little cruise I pre paid for, just didn't feel like going. I knew I disliked traveling and this trip was a massive reminder as to why. About the only thing that was a positive surprise for me was the long haul flights. I was dreading it at first but the nice reclining seats and big screen LCDs allowed me to kick back and stretch my legs without getting the usual cramps and discomfort. My flight to Amsterdam was on a single airline, made a stop in Chicago where the transfer of plans was amazingly short - it was about 150 feet between the gates I was afraid that it was going to be far and maybe I'd miss the flight (I don't have much recent flying experience the last time I had to make a connecting flight was I'd wager 20 years ago).

I got confused as to my flight schedule(wasn't going to be the first time) and I arrived in Amsterdam about eight hours before I thought I was going to arrive. The hotel was alright, I mean for the price at least, it was around $200/night or something which seemed pretty typical for a city room. First thing I noticed is it took me a good 3-5 minutes to figure out how the lights worked (had to put hotel key card in a slot to activate them). Took a shower after the long flight - no washcloth ? Maybe it is not typical in Europe I don't know, I seem to recall washcloths at hotels I was at in Asia growing up. The toilet was a very strange design, it was like this, which had a couple drawbacks. The mini bar in the room was automatic, I didn't notice that until the 2nd day, so you can't even take something out to look at it without being charged. I ended up taking quite a bit of things out. There was a sort of mini mart at a shell station about a half mile away that I walked to to buy drinks and stuff on a couple occasions the selection paled in comparison to similar stores in the U.S. The first time I went I literally saw a line of cars at the pumps. I don't know if gas was cheap or if it was a rush hour or the only gas station in the area but  it really reminded me of seeing the pictures of the gas shortages in the 70s in the U.S. There wasn't many pumps - I think 4 or 5, I'd say less than half the typical gas station here.

On the first leg of my flight the passenger next to me said watch out for the bikes - but didn't elaborate. Wow - I had not seen so many bikes since I lived in China in 89-90. They certainly have their bike infrastructure laid out pretty well with dedicated pathways for bikes as well as dedicated street crossing signals etc. On one of my walks around the hotel area I walked through what appeared to be their version of the Park and Ride. Where here the park and rides are filled with cars and parking lots, this one was filled with bikes and was pretty much entirely under a freeway overpass. It seemed like a large number of bikes weren't even locked up. The overall quality of the bikes seemed low I suppose that is at least partly to reduce theft by not having nice fancy bikes I'm not sure. More than anything when I saw the bike stuff it made me think this must be what those hippies in Seattle and SFO want. It was certainly an interesting design, too much of a culture shock for me though.

I found the intersections very confusing and am even more glad I did not try to rent a car while I was there.

Speaking of cars, wow are they small over there, I struggle to think of seeing even a single pickup truck (of any size) while I was there. I saw a bunch of cars like mine, and there was this other really tiny car, which made those tiny Smart cars look big, it was smaller than a golf cart. I missed a few opportunities to take pictures of them, I'm sure I could find them online somewhere. The taxi drivers drove sort of crazy, drifting between lanes and stuff, one of them blew way through a red light(the other lights must've turned green already) which was freaky. I recall on that same trip we were behind some kind of small van that had a radiation warning sign on it.

The data center was -- interesting I guess. Everyone had to wear protective booties around their shoes while on the floor which was a first for me, I think way over kill. Nothing really exciting here, I got everything done that I needed to get done.

I spent hours looking online for places to go but could not find anything that I was interested in. Well there was one thing I just couldn't figure out how to do it. I was really interested in seeing the big water structures they used to hold back the water. The biggest of them appeared to be a 2 hour drive away from the city ( too far). There was a couple tours that hit them but they were minimum 8 hour commitment which was too long. This is my first trip where I did not have a car at the destination and that was a good reason why I didn't do anything or go anywhere, normally I would just roam around but relying on taxis I really had to have a precise destination. I wasn't about to rent a car, I really did not feel anywhere comfortable enough to drive in a foreign country like that. While everyone said "they all speak english!", most people did speak great english, but the destinations for me for the most part were unpronounceable and not understandable (Schepenbergweg was the street the data center was on - I heard it pronounced at least a dozen times and at the end was no closer to beginning to pronounce it myself than hearing it the first time). Because of the $20 per megabyte roaming data fees on my phone I kept the data services on the phone disabled throughout my trip there which of course limited my ability to find stuff while not at the hotel or data center. I was especially worried of getting lost and having to call for a taxi and not be able to pronounce where I was and the taxi not being able to find me. I don't know how it was like in the real down town parts of town but in all the places I visited while growing up in Asia there was taxis everywhere you could just flag down and get one. I did not see this situation in the areas I was at in Amsterdam. The hotel called me a taxi to go to the data center and I asked the security guards at the data center to call me a taxi to get back.

So in the end I ate most of my meals at the hotel, never went to the down town part of town, I walked around a bit around the hotel and took some pictures of the area, nothing special. It really reminded me how much I dislike traveling in general.

The flight back was a little more frustrating, having to stop in London and go through customs and immigration and a pretty long trip to change terminals, it seems like I barely made the flight despite having a 2 hour stop over. I had to ask multiple people for help while there too because while I had a boarding pass it didn't tell me which gate or even which terminal to go to. Even once I knew where to go, getting there wasn't clear either. The whole place was very confusing, and as a result very frustrating.

This is the first trip I've taken in recent memory where I was really excited about going home. I wasn't looking forward to it to begin with and it turned out about the way I expected. Hopefully that's my last trip for a long time to come.

I thought about going somewhere fancy to eat or something, but I really couldn't find anything of interest. Add to that I don't like going out alone, if I'm with a friend things are different. When it comes to things like fancy steak or pasta or whatever I really don't have the sensitivity to tell the difference between most of them so I wouldn't be able to appreciate the good stuff so there really isn't a whole lot of point of me going. There was a BBQ + Grill near the data center (emphasis on was), the sign was still up but the building was empty. I went to two different nice places with a local friend when I was in Atlanta that I really enjoyed, I tried finding something sort of along those lines in Amsterdam but came up with nothing. Most of the places seemed too exotic or too fancy/upper class.

Apparently I left on the day things were going to get crazy, some special soccer game was being played on Saturday afternoon (I left at around noon). I've never been much of a soccer fan at least not since I played it back in 5th grade and earlier years. About the only sport I do enjoy watching is pro football, and even then my interest has been waning over the recent years.

I did all of my shopping at the Airport, picked up a bunch of dutch chocolate going to give most of it away, I tried some of it and it tastes like regular chocolate. I live a mile or so away from a pretty big Sees Candy operation, I bought some of their stuff for Christmas gifts last year, it tastes similar to the Dutch stuff if not better. Picked up a couple picture books of the area, along with some shot glasses for friends and/or family or something.

I got back a full day earlier than I expected. I was absolutely sure yesterday was Monday when I woke up at 5:30AM and turned to CNBC only to see it was Sunday. I got back on Saturday afternoon.

Contrast that with my next trip, which I think will be early July at this point, road trip up to Seattle. I decided to take the coast up north at least to Crescent City, CA. I've been wanting to take my new car along the coast since I bought it over a year ago. I made the coastal trip a couple of times several years ago but not in a car as fun to drive as the one I have at the moment. I'm not sure if I will spend two or three days driving up. I'm really looking forward to that. I think it may of been really cool to go along the coast of the Netherlands but I really didn't have a way to make that happen while I was there.

One of my friends from SEA is in town for a few days I intend to take tomorrow off and go see him down in Morgan Hill, CA (60 miles away), should be good times to catch up and hang out at this nice place he is talking about.

Tagged as: Comments Off