TechOpsGuys.com Diggin' technology every day

June 11, 2012

3PAR and NPIV

Filed under: Storage — Tags: , — Nate @ 7:20 am

I was invited to a little preview of some of the storage things being announced at HP Discover last week, just couldn’t talk about it until the announcement. Since I was busy in Amsterdam all last week I really didn’t have a lot of time to think about blogging here.

But I’m back and am mostly adjusted to the time zone differences I hope. HP had at least two storage related announcements they made last Monday, one related to scaling of their StoreOnce dedupe setup and another related to 3PAR. The StoreOnce announcement seemed to be controversial, since I really have a minimal amount of exposure to that sort of product I won’t talk about it much, on the surface it sounded pretty impressive but if the EMC claims are true than it’s unfortunate.

Anyways onto the 3PAR announcement which while it had a ton of marketing around it, it basically comes down to three words:

3PAR Supports NPIV (finally)

NPIV in a nutshell the way I understand it is a way of virtualizing connections between points in a fibre channel network, most often in the past it seems to have been used to present storage directly to VM hosts, via FC switches. NPIV is also used by HP’s VirtualConnect technology on the FC side to connect the VC modules to a NPIV-aware FC switch (which is pretty much all of them these days?), and then the switch connected to the storage(duh). I assume that NPIV is required by Virtual Connect because the VC module isn’t really a switch it’s more of a funky bridge.

Because 3PAR did not support NPIV (for what reason I don’t know I kept asking them about it for years but never got a solid response as to why not or when they might support it) there was no way to directly connect a Virtual Connect module (either the new Flex Fabric or the older dedicated FC VC modules) to a 3PAR array, you had to have a switch as a middleman. Which just seemed like a waste. I mean here you have a T or now a V-class system with tons of ports, you have these big blade chassis with a bunch of servers in them, with the VC modules acting like a switch (acting as in aggregating points) and you can’t directly connect it to the 3PAR storage! It was an unfortunate situation. Even going back to the 3cV, which was a bundle of sorts of 3PAR, HP c-Class Blades and VMware (long before HP bought 3PAR of course), I would have thought getting NPIV support would of been a priority but it didn’t happen, until now (well last Monday I suppose).

So at scale you have up to 96 host fibre channel ports on a V400 or 192 FC ports on a V800 operating at 8Gbps. At a maximum you could get by with 48 blade enclosures (2 FC/VC modules each with a single connection) on a V400 or of course double that to 96 on a V800. Cut it in half if you want higher redundancy with dual paths on each FC/VC module. That’s one hell of a lot of systems directly connected to the array. Users may wish to stick to a single connection per VC module allowing the 2nd connection to be connected to something else, maybe another 3PAR array. You still have full redundancy with two modules and one path per module. 3PAR 4Gbps HBAs (note the V-class has 8Gbps) have queue depths of something like 1,536 (not sure what the 8Gbps HBAs have). If your leveraging full height blades you get 8 per chassis, absolute worst case scenario you could set a queue depth of 192/server (I use 128/server on my gear). You could probably pretty safely go quite a bit higher though more thought may have to be had in certain circumstances. I’ve found 128 has been more than enough for my own needs.

It’s cost effective today to easily get 4TB worth of memory per blade chassis, memory being the primary driver of VM density, so your talking anywhere from 96 – 384 TB of memory hooked up to a single 3PAR array. From a CPU perspective anywhere from 7,680 CPU cores all the way up to 36,684 CPU cores in front of a single storage system, a system that has been tested to run at over 450,000 SPC-1 IOPS. The numbers are just insane.

All we need now is a flat ethernet fabric to connect the Virtual Connect switches to, oh wait we have that too, though it’s not from HP. A single pair of Black Diamond X-Series switches could scale to the max here as well, supporting a full eight 10Gbit/second connections per blade chassis with 96 blade chassis directly connected – which, guess what – is the maximum number of 10GbE ports on a pair of FlexFabric Virtual Connect modules (assuming your using two ports for FC). Of course all of the bandwidth is non blocking. I don’t know what the state of interoperability is but Extreme touts their VEPA support in scaling up to 128,000 VMs in an X-series, and Virtual Connect appears to tout their own VEPA support as well. Given the lack of more traditional switching functionality in the VC modules it would probably be advantageous to leverage VEPA (whether or not this extends to the Hypervisor I don’t know – I suspect not based on what I last heard at least from VMware, I believe it is doable in KVM though) to route that inter-server traffic through the upstream switches in order to gain more insight into it and even control it. If you have upwards of 80Gbps of connectivity per chassis anyways it seems there’d be abundant bandwidth to do it. All HP needs to do now is follow the Dell and revise their VC modules to natively support 40GbE (the Dell product is a regular Blade Ethernet switch by contrast and is not yet shipping).

You’d have to cut at least one chassis out of that configuration(or reduce port counts) in order to have enough ports on the X-Series to uplink to other infrastructure. (When I did the original calculations I forgot there would be two switches not one, so there’s more than enough ports to support 96 blade chassis between a pair of X-8s going full bore with 8x10GbE/chassis and you could even use M-LAG to go active-active. if you prefer). I’m thinking load balancers, and some sort of scale-out NAS for file sharing, maybe the interwebs too.

Think about that, up to 30,000 cores, more than 300 TB of memory, sure you do have a bunch of bridges, but all of it connected by only two switches, and one storage array (perhaps two). Just insane.

One HP spokesperson mentioned that even a single V800 isn’t spec’d to support their maximum blade system configuration of 25,000 VMs. 25k VMs on a single array does seem quite high(that comes to an average of 18 SPC-1 IOPS/VM), but it really depends on what those VMs are doing. I don’t see how folks can go around tossing solutions about saying X number of VMs when workloads and applications can vary so widely.

So in short, the announcement was simple – 3PAR supports NPIV now – the benefits of that simple feature addition are pretty big though.

Back from Amsterdam

Filed under: Uncategorized — Tags: — Nate @ 5:20 am

I’m back from Amsterdam – it was about what I expected. I basically stuck to the hotel and the data center – I even skipped out on that little cruise I pre paid for, just didn’t feel like going. I knew I disliked traveling and this trip was a massive reminder as to why. About the only thing that was a positive surprise for me was the long haul flights. I was dreading it at first but the nice reclining seats and big screen LCDs allowed me to kick back and stretch my legs without getting the usual cramps and discomfort. My flight to Amsterdam was on a single airline, made a stop in Chicago where the transfer of plans was amazingly short – it was about 150 feet between the gates I was afraid that it was going to be far and maybe I’d miss the flight (I don’t have much recent flying experience the last time I had to make a connecting flight was I’d wager 20 years ago).

I got confused as to my flight schedule(wasn’t going to be the first time) and I arrived in Amsterdam about eight hours before I thought I was going to arrive. The hotel was alright, I mean for the price at least, it was around $200/night or something which seemed pretty typical for a city room. First thing I noticed is it took me a good 3-5 minutes to figure out how the lights worked (had to put hotel key card in a slot to activate them). Took a shower after the long flight – no washcloth ? Maybe it is not typical in Europe I don’t know, I seem to recall washcloths at hotels I was at in Asia growing up. The toilet was a very strange design, it was like this, which had a couple drawbacks. The mini bar in the room was automatic, I didn’t notice that until the 2nd day, so you can’t even take something out to look at it without being charged. I ended up taking quite a bit of things out. There was a sort of mini mart at a shell station about a half mile away that I walked to to buy drinks and stuff on a couple occasions the selection paled in comparison to similar stores in the U.S. The first time I went I literally saw a line of cars at the pumps. I don’t know if gas was cheap or if it was a rush hour or the only gas station in the area but  it really reminded me of seeing the pictures of the gas shortages in the 70s in the U.S. There wasn’t many pumps – I think 4 or 5, I’d say less than half the typical gas station here.

On the first leg of my flight the passenger next to me said watch out for the bikes – but didn’t elaborate. Wow – I had not seen so many bikes since I lived in China in 89-90. They certainly have their bike infrastructure laid out pretty well with dedicated pathways for bikes as well as dedicated street crossing signals etc. On one of my walks around the hotel area I walked through what appeared to be their version of the Park and Ride. Where here the park and rides are filled with cars and parking lots, this one was filled with bikes and was pretty much entirely under a freeway overpass. It seemed like a large number of bikes weren’t even locked up. The overall quality of the bikes seemed low I suppose that is at least partly to reduce theft by not having nice fancy bikes I’m not sure. More than anything when I saw the bike stuff it made me think this must be what those hippies in Seattle and SFO want. It was certainly an interesting design, too much of a culture shock for me though.

I found the intersections very confusing and am even more glad I did not try to rent a car while I was there.

Speaking of cars, wow are they small over there, I struggle to think of seeing even a single pickup truck (of any size) while I was there. I saw a bunch of cars like mine, and there was this other really tiny car, which made those tiny Smart cars look big, it was smaller than a golf cart. I missed a few opportunities to take pictures of them, I’m sure I could find them online somewhere. The taxi drivers drove sort of crazy, drifting between lanes and stuff, one of them blew way through a red light(the other lights must’ve turned green already) which was freaky. I recall on that same trip we were behind some kind of small van that had a radiation warning sign on it.

The data center was — interesting I guess. Everyone had to wear protective booties around their shoes while on the floor which was a first for me, I think way over kill. Nothing really exciting here, I got everything done that I needed to get done.

I spent hours looking online for places to go but could not find anything that I was interested in. Well there was one thing I just couldn’t figure out how to do it. I was really interested in seeing the big water structures they used to hold back the water. The biggest of them appeared to be a 2 hour drive away from the city ( too far). There was a couple tours that hit them but they were minimum 8 hour commitment which was too long. This is my first trip where I did not have a car at the destination and that was a good reason why I didn’t do anything or go anywhere, normally I would just roam around but relying on taxis I really had to have a precise destination. I wasn’t about to rent a car, I really did not feel anywhere comfortable enough to drive in a foreign country like that. While everyone said “they all speak english!”, most people did speak great english, but the destinations for me for the most part were unpronounceable and not understandable (Schepenbergweg was the street the data center was on – I heard it pronounced at least a dozen times and at the end was no closer to beginning to pronounce it myself than hearing it the first time). Because of the $20 per megabyte roaming data fees on my phone I kept the data services on the phone disabled throughout my trip there which of course limited my ability to find stuff while not at the hotel or data center. I was especially worried of getting lost and having to call for a taxi and not be able to pronounce where I was and the taxi not being able to find me. I don’t know how it was like in the real down town parts of town but in all the places I visited while growing up in Asia there was taxis everywhere you could just flag down and get one. I did not see this situation in the areas I was at in Amsterdam. The hotel called me a taxi to go to the data center and I asked the security guards at the data center to call me a taxi to get back.

So in the end I ate most of my meals at the hotel, never went to the down town part of town, I walked around a bit around the hotel and took some pictures of the area, nothing special. It really reminded me how much I dislike traveling in general.

The flight back was a little more frustrating, having to stop in London and go through customs and immigration and a pretty long trip to change terminals, it seems like I barely made the flight despite having a 2 hour stop over. I had to ask multiple people for help while there too because while I had a boarding pass it didn’t tell me which gate or even which terminal to go to. Even once I knew where to go, getting there wasn’t clear either. The whole place was very confusing, and as a result very frustrating.

This is the first trip I’ve taken in recent memory where I was really excited about going home. I wasn’t looking forward to it to begin with and it turned out about the way I expected. Hopefully that’s my last trip for a long time to come.

I thought about going somewhere fancy to eat or something, but I really couldn’t find anything of interest. Add to that I don’t like going out alone, if I’m with a friend things are different. When it comes to things like fancy steak or pasta or whatever I really don’t have the sensitivity to tell the difference between most of them so I wouldn’t be able to appreciate the good stuff so there really isn’t a whole lot of point of me going. There was a BBQ + Grill near the data center (emphasis on was), the sign was still up but the building was empty. I went to two different nice places with a local friend when I was in Atlanta that I really enjoyed, I tried finding something sort of along those lines in Amsterdam but came up with nothing. Most of the places seemed too exotic or too fancy/upper class.

Apparently I left on the day things were going to get crazy, some special soccer game was being played on Saturday afternoon (I left at around noon). I’ve never been much of a soccer fan at least not since I played it back in 5th grade and earlier years. About the only sport I do enjoy watching is pro football, and even then my interest has been waning over the recent years.

I did all of my shopping at the Airport, picked up a bunch of dutch chocolate going to give most of it away, I tried some of it and it tastes like regular chocolate. I live a mile or so away from a pretty big Sees Candy operation, I bought some of their stuff for Christmas gifts last year, it tastes similar to the Dutch stuff if not better. Picked up a couple picture books of the area, along with some shot glasses for friends and/or family or something.

I got back a full day earlier than I expected. I was absolutely sure yesterday was Monday when I woke up at 5:30AM and turned to CNBC only to see it was Sunday. I got back on Saturday afternoon.

Contrast that with my next trip, which I think will be early July at this point, road trip up to Seattle. I decided to take the coast up north at least to Crescent City, CA. I’ve been wanting to take my new car along the coast since I bought it over a year ago. I made the coastal trip a couple of times several years ago but not in a car as fun to drive as the one I have at the moment. I’m not sure if I will spend two or three days driving up. I’m really looking forward to that. I think it may of been really cool to go along the coast of the Netherlands but I really didn’t have a way to make that happen while I was there.

One of my friends from SEA is in town for a few days I intend to take tomorrow off and go see him down in Morgan Hill, CA (60 miles away), should be good times to catch up and hang out at this nice place he is talking about.

June 1, 2012

London Internet Exchange downed by Loop

Filed under: Networking — Tags: , — Nate @ 8:08 am

This probably doesn’t happen very often at these big internet exchanges but found the news sort of interesting.

I had known  for a few years that the LINX was a dual vendor environment, one side was Foundry/Brocade the other was Extreme, they are one of the few places that go out of their way to advertise what they use. I’m sure it gets them a better discount :)  It seems the LINX replaced the Foundry/Brocade with Juniper at some point since I last checked(less than a year ago). Though their site still mentions usage of EAPS (Extreme’s ring protocol) and MRP (Foundry’s ring protocol). I assume Juniper has not adopted MRP, though they probably have something similar. Looking at the design of the Juniper LAN vs the Extreme LAN (and the Brocade LAN before Juniper), the Juniper one looks a lot more complicated.  I wonder if they are using Juniper’s new protocol(s) to manage it? Qfabric I think it’s called? It seems LINX still has some Brocade in one of their edge networks.

Apparently the Juniper side is what suffered the loop –

“Linx is trying to determine where the loop originated and we are also addressing why the protection on Juniper’s LAN didn’t work.”

I wanted to point out again, since it’s been a while since I covered it (and only then was it buried in the post, wasn’t part of the title), that Extreme has a protocol (that as far as I know is unique – let me know if there is another vendor or protocol that is similar – note of course I am not referring to anything like STP) that can detect and recover(in some cases) loops automatically. I’ve only used it in detect mode to-date. I was also telling someone about this protocol who was learning the ropes on Extreme gear after coming from a Juniper background so thought I would mention it again.

The protocol is the Extreme Loop Recovery Protocol (ELRP). The documentation does a better job at explaining it than I can.

The Extreme Loop Recovery Protocol (ELRP) is used to detect network loops in a Layer 2 network. A switch running ELRP transmits multicast packets with a special MAC destination address out of some or all of the ports belonging to a VLAN. All of the other switches in the network treat this packet as a regular, multicast packet and flood it to all of the ports belonging to the VLAN.

When the packets transmitted by a switch are received back by that switch, this indicates a loop in the Layer 2 network. After a loop is detected through ELRP, different actions can be taken such as blocking certain ports to prevent loop or logging a message to system log. The action taken is largely dependent on the protocol using ELRP to detect loops in the network.

The design seems simple enough to me, I’m not sure why others haven’t come up with something similar (or if they have let me know!)

It’s rare to have a loop in a data center environment but I do remember a couple loops I came across in an office environment many years ago that ELRP helped trace down. I’m not sure what method one would use to trace down a loop without something like ELRP – perhaps just looking at port stats and trying to determine where the bulk of the traffic is and disabling ports or unplugging cables until it stops.

[Tangent]

I remember an outage one company I was at took one time to upgrade some of our older 10/100 3COM switches to gigabit Extreme switches. It was a rushed migration, I was working with the network engineer that we had, the switches were installed in a centralized location with tons of cables, none of which were labeled. So I guess it comes as little surprised while during the migration someone (probably me) happened to plug the same cable back into one of the switches causing a loop. It took a few minutes to track down, at one point our boss was saying get ready to roll back. The network engineer and I looked at each other and laughed there was no roll back, well not one that was going to be smooth it would of taken another hour of downtime to remove the Extreme switches and re-install the 3COM and re-cable stuff. Fortunately I found the loop. This was about a year or so before I was aware of the existence of ELRP. We discovered the loop mainly after all the switch lights started blinking in sequence, normally a bad thing. Then users reported they lost all connectivity.

One of my friends who is another network engineer told me a story when I was in Atlanta earlier in the year about a customer who was a university or something. They had major network performance problems but could not track them down. These problems had been going on for literally months. My friend went out as a consultant and they brought him into their server/network room and his jaw dropped, they had probably 2 dozen switches and ALL of them were blinking in sequence. He knew what the problem was right away and informed the customer. But the customer was adamant that the lights were supposed to blink that way and the problem was elsewhere(not kidding here). The customer had other issues like running overlapping networks on the same VLAN etc. My friend had a lot of suggestions for the customer but the customer felt insulted by him telling them their network had so many problems so they kicked him out and told the company not to send him back. A couple months later the customer went through some sort of audit process and failed miserably and grudgingly asked (begged) to get my friend back since he was the only one they knew that seemed to know what he was doing. He went back and fixed the network I assume (I forgot that last bit of the story).

[End Tangent]

ELRP can detect a loop immediately and give a very informative system log entry as to the port(s) the loop is occurring on so you can take action. It works best of course if it is running on all ports, so you can pinpoint down to the edge port itself. But if for some reason the edge is not an Extreme switch at least you can get it at a higher layer and can isolate it further that way.

You can either leave it running periodically every X seconds it will send a probe out, or you can run it on demand for a real time assessment. There is also integration with ESRP which I wrote about a while ago, although I don’t use the integrated mode (see the original post as to how that works and why). I normally leave it running sending requests out at least say once every 30 seconds.

LINX had another outage (which was the last time I looked at their vendor stats) a couple of years ago (this one affected me since my company had gear hosted in London at the time and our monitors were tripped by this event), though no mention of which LAN the outage occurred on. One user wrote

It wasn’t a port upgrade, a member’s session was being turned up and due to miscommunication between the member’s engineer and the LINX engineer a loop was somehow introduced in to the network which caused a broadcast storm and a switches CPU to max out cue packet loss and dropped BGP sessions.

As a cause to the outage that occurred two years ago. So I guess it was another loop! For all I know LINX is not running ELRP in their environment either.

It’s not exactly advertised by Extreme if you talk to them, it’s one of those things that’s buried in the docs. Same goes for ESRP. Two really useful protocols that Extreme almost never mentions, two things that make them stand out in the industry and they don’t talk about them. I’m told that one reason could be is they are proprietary(vs EAPS which is not and Extreme touts EAPS a lot but EAPS is layer 2 only!), though as I have mentioned in the past ESRP doesn’t require any software at the edge to function and can support managed and unmanaged devices. So you don’t require an Extreme-only network to run (just at the core, like most any other protocol). ELRP is even less stringent – can be run on any Extreme switch, no interoperability issues. If there were open variants of the protocols that’d be better of course, but again, these seem to be unique in the industry so tout what you got! Customers don’t have to use them if they don’t want to and it can make a network administrator’s life vastly simpler in many cases by leveraging what you have available to you. Good luck integrating Extreme or Cisco or Brocade into Juniper’s Qfabric ? Or into Force10’s distributed core setup ? There are interoperability issues abound with most of the systems out there.

« Newer Posts

Powered by WordPress