I remember reading a while ago how Netgear teamed up with Extreme to re-sell their Black Diamond series of switches.

I didn’t think too much of it till I was on Netgear’s site today so I decided to try to poke around and see if I could find the product(s) that were being resold or OEM’d, and I found them, here is one.

The Netgear 8800

When I saw that it just looked so strange! It’s HP-grey in color, not the usual purple I’m used to seeing. Speaking of HP and purple someone at HP recently speculated to me that the 3PAR arrays will likely stick to being yellow instead of HP-grey because it makes them stand out in the data center.

Tangent comin’ hold onto your butts…

While troubleshooting my home network this morning I think I let some of the smoke out of my HP workstation. Which reminds me of this quote I came across on slashdot years ago

 There is no such thing as a "safe" capacitor! They are filled with SMOKE and that smoke is DEADLY. ALWAYS let the smoke out of the capacitors before attempting to handle them! This should only be done by PROFESSIONALS. Do NOT try this at home.

 Always assume a CAPACITOR is holding a charge. And: Capacitors don't kill people, it's the circuit of which the person is a part that is dangerous...

I thought the networking issue may of been somehow caused by the HP box, so I rebooted it, while it was in the midst of rebooting(middle of POST before the screen came up), I powered it off(by holding down the power button), to reset the network chipset entirely. When I did that I heard a weird clicking sound coming from either the HP box(I think so) or my Cyberpower UPS which was right next to it. Within about 10 seconds I swear a little puff of smoke came out of the HP box(I think), there’s a remote chance it was just dust but I don’t think so. I unplugged the HP box and the clicking stopped. Then I plugged it back in about 30 seconds later, which caused it to turn on automatically, it booted like a champ, no errors, the UPS event log reported nothing. So I don’t know what inside the HP box released the smoke but I guess it was not vital?

Back on topic..

Anyways I poked around in the user manuals and they did a pretty good job of replacing all references of the original product and making it look like a Netgear product through and through (with a couple minor exceptions in diagrams).

I remember about 11 years ago now when I was shopping for a Summit 48 on Ebay for my company(this product wasn’t known for quality at the time though I didn’t know it at the time), I came across some Compaq OEM’d Summit 48s that I think were white in color.

If I was building a bigger network I really would be tempted to opt for this Netgear product if for nothing else to see the expression on people’s faces when I tell them I’m using Netgear, not a brand that comes to most people’s minds when it comes to data center networks! Speaking of data center it looks like Extreme’s 40GbE offerings are leading the market pretty good, I’m so proud of them! Hopefully they can sustain the execution and gain market share. They’ve had some missteps in the past which has knocked them back a few notches(at the time), but they certainly have another opportunity here.

I remember when HP used to OEM/re-sell Foundry Networks chassis switches though I seem to recall HP not making any modifications to the chassis itself(at least according to the pictures on the website, I don’t think it even had an HP logo on the thing). The product at the time was the MG8 which I was entertaining for a data center build out back in 2004/2005. I wasn’t going to buy from HP but was just one of those days that I was poking around and came across it on HP’s site.

Oh and in case your wondering my home network used to be powered by Extreme, I had a trusty Summit 48 for many years, which I eventually upgraded to a Summit 48si (which I still have now). I stopped using it many years ago because I just didn’t have enough ports at home to justify the power usage or more importantly the noise, 1U data center switches are so noisy for home use! I went so far as to replace all of the fans in the 48si (I believe I used Sunon Maglev fans) with quieter ones which reduced the noise by at least half but it was still really loud.

The patented MagLev design is based on magnetic principles and forces that not only propel the fan but also ensure stable rotation over its entire 360 degrees of movement. Utilizing the attraction of the magnetic levitation force, MagLev eliminates the wobbling and shaking problems of traditional motor fans. With this new technology, the MagLev fan propeller is suspended in air during rotation so that the shaft and bearing do not come into direct contact with each other to create friction.

(I dig technology even when it comes to fans!)

The Summit 48 by contrast was 2U and had 80mm fans which spun slower and were quieter. At the moment I have 9 devices wired into my netgear-powered home switching network (one 8-port switch and one 5-port). I used to have a couple Foundry load balancers, and a Cisco switch and a couple other things I think but I recycled them with my Summit 48 years ago, was too lazy to try to re-sell them.

I just saw that picture and was just fascinated by it. It also gives me another opportunity to add more color onto this blog.

   Send article as PDF to

I’d like to consider myself at least somewhat up to date on what is out there so when things like this come out I do find it interesting to see what they say.

The thing that stands out to me the most: Arista Networks has only 75 employees ?!? Wow, they’ve been able to do all of that work with only 75 employees? Really? Good job.. that is very surprising to me, I mean most of the companies I have worked at have had more than 75 employees and they’ve accomplished (in my opinion) a fraction of what Arista seems to have, at least from a technology stand point (revenue wise is a different story assuming again the report is accurate).

The thing that made me scratch my head the most: Cisco allows you to run virtual machines on their top of rack switches? Huh? Sounds like EMC and them wanting you to run VMs on their VMAX controllers? I recall at one point Citrix and Arista teamed up to allow some sort of VM to run Netscaler embedded in the Arista switches, though never heard of anyone using it and never heard Citrix promoting it over their own stuff. Seemed like an interesting concept, though no real advantage to doing it I don’t think (main advantage I can think of is non blocking access to the switch fabric which really isn’t a factor with lower end load balancers since they are CPU bound not network bound).

The report seems to take a hypothetical situation where a fairly large organization is upgrading their global network and then went to each of the vendors and asked for a proposal. They leave out what each of the solutions was, specifically which is dissapointing.

They said HP was good because it was cheap, which is pretty much what I’ve heard in the field, it seems nobody that is serious runs HP Procurve.

They reported that Juniper and Brocade were the most “affordable” (having Juniper and affordable together makes no sense), and Arista and Force10 being least affordable (which seems backwards too – they are not clear on what they used to judge costs, because I can’t imagine a Force10 solution costing more than a Juniper one).

They placed some value on line cards that offered both copper and fiber at the same time, which again doesn’t make a lot of sense to me since you can get copper modules to plug into SFP/SFP+ slots fairly easily. The ability to “Run VMs on your switch” also seemed iffy at best, they say you can run “WAN optimization” VMs on the switches, which for a report titled “Data center networking” really should be a non issue as far as features go.

The report predicts Brocade will suffer quite a bit since Dell now has Force10. How Brocade doesn’t have as competitive products as they otherwise could have.

They tout Juniper’s ability to have multiple power supplies, switch fabrics, routing modules as if it was unique to Juniper, which makes no sense to me either. They do call out Juniper for saying their 640-port 10GbE switch is line rate only to 128 ports.

They believe Force10 will be forced into developing lower end solutions to fill out Dell’s portfolio rather than staying competitive on the high end, time will tell.

Avaya? Why bother? They say you should consider them if you’ve previously used Nortel stuff.

They did include their sample scenario that they sent to the vendors and asked for solutions for. I really would of liked to have seen the proposals that came back.

A four-site organization with 7850 employees located at a Canadian head office facility, and three branch offices located in the US, Europe, and Canada. The IT department consists of 100 FTE, and are located primarily at the Canadian head office, with a small proportion of IT staff and systems located at the branch offices.

The organization is looking at completing a data center refurbish/refresh:

The organization has 1000 servers, 50% of which are virtualized (500 physical). The data center currently contains 40 racks with end-of-row switches. None of the switching/routing layers have any redundancy/high availability built in, leaving many potential single points of failure in the network (looking for 30% H/A).

A requirements gathering process has yielded the need for:

• A redundant core network, with capacity for 120 x 10Gbps SFP+ ports
• Redundant top of rack switches, with capacity for 48 x 1Gbps ports in each rack
• 1 ready spare ToR switch and 1 ready spare 10Gps chassis card
• 8x5xNBD support & maintenance
• Nameplate data – power consumption – watts/amps
• 30% of the servers to be highly available

It is unclear how redundant they expect the network to be, would a single chassis with redundant fabrics and power supplies be enough or would you want two? They are also not clear as to what capabilities their ToR switches need other than the implied 10Gbps uplinks.

If I were building this network with Extreme gear I would start out with two pairs of stacked X670Vs at the core(each stack having 96x10GbE), each stack would be connected by 2x40GbE connections with passive copper cabling. The two stacks would be linked together with 4x40GbE connections with passive copper cabling, running (of course) ESRP as the fault tolerance protocol of choice between the two. This would provide 192x10GbE ports between the two stacks, with half being active half being passive.

Another, simpler approach would be to just stack three of the X670V switches together for 168x10GbE active-active ports. Though you know I’m not a big fan of stacking(any more than I am running off a single chassis), if I am connecting 1000 servers I want a higher degree of fault tolerance.

Now if you really could not tolerate a active/passive network, if you really needed that extra throughput then you can use M-LAG to go active-active at layer 2, but I wouldn’t do that myself unless you were really sure you needed that ability. I prefer the reduced complexity with active/passive.

As for the edge switches, they call for redundant 48 port 1GbE switches. Again they are not clear as to their real requirements, but what I would do (what I’ve done in the past) is two stand alone 48-port 1GbE switches, each with 2x10GbE (Summit X460) or 4x10GbE(Summit X480) connections to the core. These edge switches would NOT be stacked, they would be stand alone devices. You could go lower cost with Summit X450e, or even Summit X350 though I would not go with the X350 for this sort of data center configuration. Again I assume your using these switches in an active-passive way for the most part(as in 1 server is using 1 switch at any given time), though if you needed a single server to utilize both switches then you could go the stacking approach at the edge, all depends on what your own needs are – which is why I would of liked to have seen more detail in the report. Or you could do M-LAG at the edge as well, but ESRP’s ability to eliminate loops is hindered if you link the edge switches together since there is a path in the network that ESRP cannot deal with directly (see this post with more in depth info on the how, what, and why for ESRP).

I would NOT go with a Black Diamond solution (or any chassis-based approach) unless cost was really not an issue at all. Despite this example organization having 1,000 servers it’s still a small network they propose building, and the above approach would scale seamlessly to say 4 times that number non disruptively providing sub second layer 2 and layer 3 fail over. It is also seamlessly upgradeable to a chassis approach with zero downtime (well sub second), should the organization needs grow beyond 4,000 hosts. The number of organizations in the world that have more than 4,000 hosts I think is pretty small in the grand scheme of things. If I had to take a stab at a guess I would say easily less than 10% maybe less than 5%.

So in all, an interesting report, not very consistent in their analysis, lacked some more detail that would of been nice to see, but still interesting to see someone else’s thought patterns.

   Send article as PDF to

Normally I think Tolly reports are halfway decent although they are usually heavily biased towards the sponsor (not surprisingly). This one though felt light on details. It felt like they rushed this to market.

Basically what Force10 is talking about is a distributed core architecture with their 32-port 40GbE Z9000 switches as what they call the spine(though sometimes they are used as the leaf), and their 48-port 10 GbE S4810 switches as what they call the leaf (though sometimes they are used as the spine).

They present 3 design options:

Force10 Distributed Core Design

I find three things interesting about these options they propose:

• The minimum node count for spine is 4 nodes
• They don’t propose an entirely non blocking fabric until you get to “large”
• The “large” design is composed entirely of Z9000s, yet they keep the same spine/leaf configuration, whats keeping them from being entirely spine?

The distributed design is very interesting, though it would be a conceptual hurdle I’d have a hard time getting over if I was in the market for this sort of setup. It’s nothing against Force10 specifically I just feel safer with a less complex design (I mentioned before I’m not a fan of stacking for this same reason), less things talking to each other in such a tightly integrated fashion.

That aside though a couple other issues I have with the report is while they do provide the configuration of the switches (that IOS-like interface makes me want to stab my eyes with an ice pick) – I’m by no means familiar with Force10 configuration and they don’t talk about how the devices are managed. Are the spine switches all stacked together? Are the spine and leaf switches stacked together? Are they using something along the lines of Brocade’s VCS technology? Are the devices managed independently and they are relying on other protocols like MLAG? The web site mentions using TRILL at layer 2, which would be similar to Brocade.

The other issue I have with the report is the lack of power information, specifically would be interested (slightly, in the grand scheme of things I really don’t think this matters all that much) in the power per usable port (ports that aren’t being used for up links or cross connects). They do rightly point out that power usage can vary depending on the workload and so it would be nice to get power usage based on the same workload. Though conversely it may not matters as much, looking at the specs for the Extreme X670V (48x10GbE + 4x40GbE) says there is only 8 watts of difference between (that particular switch) 30% traffic load and 100% traffic load, seems like a trivial amount.

Extreme Networks X670V Power Usage

As far as I know the Force10 S4810 switch uses the same Broadcom chipset as the X670V.

On their web site they have a nifty little calculator where you input your switch fabric capacity and it spits out power/space/unit numbers. The numbers there don’t sound as impressive:

• 10Tbps fabric = 9.6kW / 12 systems / 24RU
• 15Tbps fabric = 14.4kW / 18 systems / 36RU
• 20Tbps fabric = 19.2kW / 24 systems / 48RU

The aforementioned many times Black Diamond X-Series comes in at somewhere around 4kW (well if you want to be really conservative you could say 6.1kW assuming 8.1W/port which their report was likely high considering system configuration) and a single system to get up to 20Tbps of fabric(you could perhaps technically say it is has 15Tbps of fabric since the last 5Tbps is there for redundancy, 192 x 80Gbps = 1.5Tbps). 14.5RU worth of rack space too.

Dell claims non-blocking scalability up to 160Tbps, which is certainly a lot! Though I’m not sure what it would take for me to make the leap into a distributed system such as TRILL. Given TRILL is a layer 2 only protocol (which I complained about a while ago), I wonder how they handle layer 3 traffic, is it distributed in a similar manor? What is the performance at layer 3? Honestly I haven’t read much on TRILL at this point (mainly because it hasn’t really interested me yet), but one thing that is not clear to me(maybe someone can clarify), is is TRILL just a traffic management protocol or does it also include more transparent system management(e.g. manage multiple devices as one), or does that system management part require more secret sauce by the manufacturer.

My own, biased(of course), thoughts on this architecture, while innovative:

• Uses a lot of power / consumes a lot of space
• Lots of devices to manage
• Lots of connections – complicated physical network
• Worries over resiliency of TRILL (or any tightly integrated distributed design – getting this stuff right is not easy)
• On paper at least seems to be very scalable
• The Z9000 32-port 40GbE switch certainly seems to be a nice product from a pure hardware/throughput/formfactor perspective. I just came across Arista’s new 1U 40GbE switch and I think I’d prefer the Force10 design with twice the size and twice the ports purely for more line rate ports in the unit.

It would be interesting to read a bit more in depth about this architecture.

I wonder if this is going to be Force10s approach going forward, the distributed design, or if they are going to continue to offer more traditional chassis products for customers who prefer that type of setup. In theory it should be pretty easy to do both.

   Send article as PDF to

Up until now, most (all?) chassis switches relied on backplanes, or more modern systems used mid planes to transmit their electrical signals between the modules in the chassis.

Something I learned a couple of years ago (I’m not an electrical engineer) is that there are physical limits as to how fast you can push those electrons over those back and mid planes. There are serious distance limitations which makes the engineering ever more complicated the faster you push the system. Here I was thinking just crank up the clock speeds and make it go faster, but apparently it doesn’t work quite that way

For the longest time all of Extreme’s products were backplane based. Then they released a mid plane based product the Black Diamond 20808 a couple of years ago. This product was discontinued earlier this year when the X-series was announced. The 20808 had (in my simple mind) a similar design to what Force10 had been doing for many years – which is basically N+1 switch fabric modules (I believe the 20808 could go to something like 5 fabric modules), all of their previous switches had what they called MSMs, or Management Switch Modules. These were combination switch fabric and management modules., with a maximum of two per system, each providing half of the switch’s fabric capacity. Some other manufacturers like Cisco separated out their switch fabric from their management module. Having separate fabric modules really doesn’t buy you much when you only have two modules in the system. But if your architecture can go to many more (I seem to recall Force10 at one point having something like 8), then of course you can get faster performance. Another key point in the design is having separate slots for your switch fabric modules so they don’t consume space that would otherwise be used by ethernet ports.

Anyways, on the Black Diamond 20808 they did something else they had never done before, they put modules on both the front, AND on the back of the chassis. On top of that the modules were criss-crossed. The modules on the front were vertical, the modules on the back were horizontal. This is purely guessing here but I speculate the reason for that is, in part, to cut the distance needed to travel between the fabric and the switch ports. HP’s c-Class Blade enclosure has a similar mid plane design with criss crossed components. Speaking of which, I wonder if the next generation 3PAR will leverage the same “non stop” midplane technology of the c-Class. The 5 Terabits of capacity on the c-Class is almost an order of magnitude more than what is even available on the 3PAR V800. Whether or not the storage system needs that much fabric is another question.

Black Diamond 20808 (rear)

The 20808 product seemed to be geared more towards service providers and not towards high density enterprise or data center computing(if I remember right the most you could get out of the box was 64x10GbE ports which you can now get in a 1U X670V).

Black Diamond 20808 (front)

Their (now very old) Black Diamond 8000 series (with the 8900 model which came out a couple of years ago being the latest incarnation) has been the enterprise workhorse for them for many years, with a plethora of different modules and switch fabric options. The Black Diamond 8900 is a backplane based product.  I remember when it came out too – it was just a couple months after I bought my Black Diamond 10808s, in the middle of 2005. Although if I remember right the Black Diamond 8800, as it was originally released, did not support the virtual router capability that the 10808 supported that I intended to base my network design on. Nor did it support the Clear Flow security rules engine. Support for these features was added years later.

You can see the impact distance has on the Black Diamond 8900 for example, with the smaller 6-slot chassis getting at least 48Gbps more switching capacity per line card than the 10-slot chassis simply because it is smaller. Remember this is a backplane designed probably seven years ago, so it doesn’t have as much fabric capacity as a modern mid plane based system.

Anyways, back on topic, the Black Diamond X-series. Extreme’s engineers obviously saw the physics (?) limits they were likely going to hit when building a next generation platform and decided to re-think how the system works, resulting, in my opinion a pretty revolutionary way of building a switch fabric (at least I’m not aware of anything else like it myself). While much of the rest of the world is working with mid planes for their latest generation of systems, here we have the Direct Orthogonal Data Path Mating System or DOD PMS (yeah, right).

Black Diamond X-Series fabric

What got me started down this path, was I was on the Data Center Knowledge web site, and just happened to see a Juniper Qfabric advertisement. I’ve heard some interesting things about Qfabric since it was announced, it sounds similar to the Brocade VCS technology. I was browsing through some of their data sheets and white papers and it came across as something that’s really complicated. It’s meant to be simple, and it probably is, but the way they explain it to me at least makes it sound really complicated. Anyways I went to look at their big 40GbE switch which is at the core of their Qfabric interconnect technology. It certainly looks like a respectable switch from a performance stand point – 128 40GbE ports, 10 Terabits of switching fabric, weighs in at over 600 pounds(I think Juniper packs their chassis products with lead weights to make them feel more robust).

So back to the report that they posted. The networking industry doesn’t have anything like the SPC-1 or SpecSFS standardized benchmarks to measure performance, and most people would have a really hard time generating enough traffic to tax these high end switches.There is standard equipment that does it, but it’s very expensive.

So, to a certain extent you have to trust the manufacturer as to the specifications of the product, a way many manufacturers try to prove their claims of performance or latency is to hire “independent” testers to run tests on the products and give reports. This is one of those reports.

Reading it made me smile, seeing how well the X-Series performed but in the grand scheme of things it didn’t surprise me given the design of the system and the fabric capacity it has built into it.

The BDX8 breaks all of our previous records in core switch testing from performance, latency, power consumption, port density and packaging design. The BDX8 is based upon the latest Broadcom merchant silicon chipset.

For the Fall Lippis/Ixia test, we populated the Extreme Networks BlackDiamond® X8 with 256 10GbE ports and 24 40GbE ports, thirty three percent of its capacity. This was the highest capacity switch tested during the entire series of Lippis/Ixia cloud network test at iSimCity to date.

We tested and measured the BDX8 in both cut through and store and forward modes in an effort to understand the difference these latency measurements offer. Further, latest merchant silicon forward packets in store and forward for smaller packets, while larger packets are forwarded in cut-through making this new generation of switches hybrid cut-through/store and forward devices.

Reading through the latency numbers, they looked impressive, but I really had nothing to compare them with, so I don’t know how good. Surely for any network I’ll ever be on it’d be way more than enough.

The BDX8 forwards packets ten to six times faster than other core switches we’ve tested.

[..]

The Extreme Networks BDX8 did not use HOL blocking which means that as the 10GbE and 40GbE ports on the BDX8 became congested, it did not impact the performance of other ports. There was no back pressure detected. The BDX8 did send flow control frames to the Ixia test gear signaling it to slow down the rate of incoming traffic flow.

Back pressure? What an interesting term for a network device.

The BDX8 delivered the fastest IP Multicast performance measured to date being able to forward IP Multicast packets between 3 and 13 times faster then previous core switch measures of similar 10GbE density.

The Extreme Networks BDX8 performed very well under cloud simulation conditions by delivering 100% aggregated throughput while processing a large combination of east-west and north-south traffic flows. Zero packet loss was observed as its latency stayed under 4.6 μs and 4.8 μs measured in cut through and store and forward modes respectively. This measurement also breaks all previous records as the BDX8 is between 2 and 10 times faster in forwarding cloud based protocols under load.

[..]

While these are the lowest Watts/10GbE port and highest TEER values observed for core switches, the Extreme Networks BDX8’s actual Watts/10GbE port is actually lower; we estimate approximately 5 Watts/10GbE port when fully populated with 768 10GbE or 192 40GbE ports. During the Lippis/Ixia test, the BDX8 was only populated to a third of its port capacity but equipped with power supplies, fans, management and switch fabric modules for full port density population. Therefore, when this full power capacity is divided across a fully populated BDX8, its WattsATIS per 10GbE Port will be lower than the measurement observed [which was 8.1W/port]

They also mention the cost of power, and the % of list price that cost is, so we can do some extrapolation. I suspect the list price of the product is not final, and I am assuming the prices they are naming are based on the configuration they are testing with rather than a fully loaded system(which as mentioned above the switch was configured with enough fabric and power for the entire chassis but only ~50% of the port capacity was installed).

Anyways, they say the price to power it over 3 years is $10,424.05 and say that is less than 1.7% of it’s list price. Extrapolating that a bit I can guesstimate that the list price of this system as tested with 352 10GbE ports is roughly $612,000, or about $1,741 per 10GbE port.

The Broadcom technology is available to the competition, the real question is how long will it take for the competition develop something that can compete with this 20 Terabit switching fabric, which seems to be about twice as fast as anything else currently on the market.

HP has been working on some next generation stuff, I read about their optical switching technology earlier this year that their labs are working on, sounds pretty cool.

[..] Charles thinks this is likely to be sometime in the next 3-to-5 years.

So, nothing on the immediate horizon on that front.

   Send article as PDF to

More than anything else this caught my eye:

On the surface it looks pretty impressive, I mean it would be interesting to see exactly how Cisco configured the competing products as in which 60 Juniper devices or 70 HP devices did they use and how were they connected?

One thing that would of been interesting to call out in such a configuration, is the number of logical devices needed for management. For example I know Brocade’s VDX product is some fancy way of connecting lots of devices sort of like more traditional stacking just at a larger scale for ease of management. I’m not sure whether or not the VDX technology extends to their chassis product as Cisco’s configuration above seems to imply using chassis switches. I believe Juniper’s Qfabric is similar. I’m not sure if HP or Arista have such technology(I don’t believe they do). I don’t think Cisco does – but they don’t claim to need it either with this big switch. So a big part of the question is managing so many devices, or just managing one. Cost of the hardware/software is one thing..

HP recently announced a revamp of their own 10GbE products, at least the 1U variety. I’ve been working off and on with HP people recently and there was a brief push to use HP networking equipment but they gave up pretty quick. They mentioned they were going to have “their version” of the 48-port 10-gig switch soon, but it turns out it’s still a ways away – early next year is when it’s supposed to ship, even if I wanted it  (which I don’t) – it’s too late for this project.

I dug into their fact sheet, which was really light on information to see what, if anything stood out with these products. I did not see anything that stood out in a positive manor, I did see this which I thought was kind of amusing -

Industry-leading HP Intelligent Resilient Framework (IRF) technology radically simplifies the architecture of server access networks and enables massive scalability—this provides up to 300% higher scalability as compared to other ToR products in the market.

Correct me if I’m wrong – but that looks like what other vendors would call Stacking, or Virtual Chassis. An age-old technology, but the key point here was the up to 300% higher scalability. Another way of putting it is at least 50% less scalable – when your comparing it to the Extreme Networks Summit X670V(which is shipping I just ordered some).

The Summit X670 series is available in two models: Summit X670V and Summit X670. Summit X670V provides high density for 10 Gigabit Ethernet switching in a small 1RU form factor. The switch supports up to 64 ports in one system and 448 ports in a stacked system using high-speed SummitStack-V160*, which provides 160 Gbps throughput and distributed forwarding. The Summit X670 model provides up to 48 ports in one system and up to 352 ports in a stacked system using SummitStack-V longer distance (up to 40 km with 10GBASE-ER SFP+) stacking technology.

In short, it’s twice as scalable as the HP IRF feature, because it goes up to 8 devices (56x10GbE each), and HP’s goes up to 4 devices (48x10GbE each — or perhaps they can do 56 too with breakout cables since both switches have the same number of physical 10GbE and 40GbE ports).

The list price on the HP switches is WAY high too, The Register calls it out at $38,000 for a 24-port switch. The X670 from Extreme has a list price of about $25,000 for 48-ports(I see it on-line for as low as about $17k). There was no disclosure of HP’s pricing for their 48-port switch.

Extreme has another 48-port switch which is cheaper (almost half the cost if I recall right – I see it on-line going for as low as $11,300) but it’s for very specialized applications where latency is really important. If I recall right they removed the PHY (?) from the switch which dramatically reduces functionality and introduces things like very short cable length limits but also slashes the latency (and cost). You wouldn’t want to use those for your VMware setup(well if you were really cost constrained these are probably better than some other alternatives especially if your considering this or 1GbE), but you may want them if your doing HPC or something with shared memory or high frequency stock trading (ugh!).

The X670 also has (or will have? I’ll find out soon) a motion sensor on the front of the switch which I thought was curious, but seems like a neat security feature, being able to tell if someone is standing in front of your switch screwing with it. It also apparently has the ability(or will have the ability) to turn off all of the LEDs on the switch when someone gets near it, and turn them back on when they go away.

(ok back on topic, Cisco!)

I looked at the Cisco slide above, and thought to myself, really, can they be that far ahead? I certainly do not go out on a routine basis and see how many devices and connectivity between them that I need to achieve  X number of line rate ports, I’ll keep it simple, if you need a large number of line rate ports just use a chassis product(you may need a few of them). It is interesting to see though, assuming it’s anywhere close to being accurate.

When I asked myself the question “Can they be that far ahead?” I wasn’t thinking of Cisco, I think I’m up to 7 readers now — you know me better than that!

I was thinking of the Extreme Networks Black Diamond X-Series which was announced (note not yet shipping…) a few months ago.

• Cisco claims to do 768 x 10GbE ports in 25U (Extreme will do it in 14.5U)
• Cisco claims to do 10W per 10GbE/port (Extreme will do it in 5W/port)
• Cisco claims to do it with 1 device .. Well that’s hard to beat but Extreme can meet them, it’s hard to do it with less than one device.
• Cisco’s new top end taps out at very respectable 550Gbit per slot (Extreme will do 1.2Tb)
• Cisco claims to do it with a list price of $1200/port. I don’t know what Extreme’s pricing will be but typically Cisco is on the very high end for costs.

Though I don’t know how Cisco gets to 768 ports, Extreme does it via 40GbE ports and breakout cables (as far as I know), so in reality the X-series is a 40GbE switch (and I think 40GbE only – to start with unless you use the break out cables to get to 10GbE).  It was a little over a year ago that Extreme was planning on shipping 40GbE at a cost of $1,000/port. Certainly the X-series is a different class of product than what they were talking about a while ago, but prices have also come down since.

X-Series is shipping “real soon now’.  I’m sure if you ask them they’ll tell you more specifics.

It is interesting to me, and kind of sad how far Force10 has fallen in the 10GbE area, I mean they seemed to basically build themselves on the back of 10GbE(or at least tried to), but I look at their current products on the very high end, and short from the impressive little 40GbE switch they have, they seem to top out at 140 line rate 10GbE in 21U. Dell will probably do well with them, I’m sure it’ll be a welcome upgrade to those customers using Procurve, uh I mean Powerconnect? That’s what Dell call(ed) their switches right?

As much as it pains me I do have to give Dell some props for doing all of these acquisitions recently and beefing up their own technology base, whether it’s in storage, or networking they’ve come a long way (more so in storage, need more time to tell in networking). I have not liked Dell myself for quite some time, a good chunk of it is because they really had no innovation, but part of it goes back to the days before Dell shipped AMD chips and Dell was getting tons of kick backs from Intel for staying an Intel exclusive provider.

In the grand scheme of things such numbers don’t mean a whole lot, I mean how many networks in the world can actually push this kind of bandwidth? Outside of the labs I really think any organization would be very hard pressed to need such fabric capacity, but it’s there — and it’s not all that expensive.

I just dug up an old price list I had from Extreme – from late November 2005. An 6-port 10GbE module for their Black Diamond 10808 switch (I had two at the time) had a list price of $36,000. For you math buffs out there that comes to $9,000 per line rate port.

That particular product was oversubscribed (hence it not being $6,000/port) as well having a mere 40Gbps of switch fabric capacity per slot, or a total of 320Gbps for the entire switch (it was marketed as a 1.2Tb switch but hardware never came out to push the backplane to those levels – I had to dig into the depths of the documentation to find that little disclosure – naturally I found it after I purchased, didn’t matter for us though I’d be surprised if we pushed more than 5Gbps at any one point!). If I recall right the switch was 24U too. My switches were 1GbE only, cost reasons

How far we’ve come..

   Send article as PDF to

It is kind of sad that Force10 was never able to pull off their IPO. I have heard that they have been losing quite a bit of talent recently, but don’t know to what degree. It’s also unfortunate they weren’t able to fully capitalize on their early leadership in the 10 gigabit arena, Arista seems to be the new Force10 in some respects, though it wouldn’t surprise me if they have a hard time growing too barring some next gen revolutionary product.

I wonder if anyone will scoop up BlueArc, they have been trying to IPO as well for a couple of years now, I’d be surprised if they can pull it off in this market.  They have good technology just a whole lot of debt. Though recently I read they started turning a profit..

   Send article as PDF to

In the early days it was mostly because of paranoia around security/encryption though over the past several years encryption has really picked up and helped that area a lot. There is still a little bit of fear in me that the encryption is not up to snuff, and I would prefer using a VPN on top of wifi to make it even more secure, only really then would I feel comfortable from a security standpoint of using wifi.

From a security standpoint I am less concerned about people intercepting my transmissions over wifi than I am about people breaking into my home network over wifi (which usually happens by intercepting transmissions – my point is more of the content of what I’m transferring, if it is important is always protected by SSL or SSH or in the case of communicating with my colo or cloud hosted server there is a OpenVPN SSL layer under that as well).

Many years ago, I want to say 2005-2006 time frame, there was quite a bit of hype around the Linksys WRT-54G wifi router, for being easy to replace the firmware with custom stuff and get more functionality out of it. So I ordered one at the time, put dd-wrt on it, which is a custom firmware that was talked a lot about back then (is there something better out there? I haven’t looked). I never ended up hooking it to my home network, just a crossover cable to my laptop to look at the features.

Then I put it back in it’s box and put it in storage.

Until earlier this week, when I decided to break it out again to play with in combination with my new HP Touchpad, which can only talk over Wifi.

My first few days with the Touchpad involved having it use my Sprint 3G/4G Mifi access point. As I mentioned earlier I don’t care about people seeing my wifi transmissions I care about protecting my home network. Since the Mifi is not even remotely related to my home network I had no problem using it for extended periods.

The problem with the Mifi, from my apartment is the performance. At best I can get 20% signal strength for 4G, and I can get maybe 80% signal strength for 3G, latency is quite bad in both cases, and throughput isn’t the best either, a lot of times it felt like I was on a 56k modem. Other times it was faster. For the most part I used 3G because it was more reliable for my location, however I do have a 5 gig data cap/month for 3G so considering I started using the Touchpad on the 1st of the month I got kind of concerned I may run into that playing with the new toy during the first month. I just checked Sprint’s site and I don’t see a way to see intra month data usage, only data usage for the month once it’s completed. The mifi tracks data usage while it is running but this data is not persisted across reboots, and I think it’s also reset if the mifi changes between 3G and 4G services. I have unlimited 4G data, but the signal strength where I’m at just isn’t strong enough.

I looked into the possibility of replacing my Mifi with newer technology, but after reading some customer reviews of the newer stuff it seemed unlikely I would get a significant improvement in performance at my location, enough to justify the cost of the upgrade at least so I decided against that for now.

So I broke out the WRT-54G access point and hooked it up. Installed the latest recommended version of firmware, configured the thing and hooked up the touchpad.

I knew there was a pretty high number of personal access points deployed near me, it was not uncommon to see more than 20 SSIDs being broadcast at any given time. So interference was going to be an issue. At one point my laptop showed me that 42 access points were broadcasting SSIDs. And that of course does not even count the ones that are not broadcasting, who knows how many there are there, I haven’t tried to get that number.

With my laptop and touchpad being located no more than 5 feet away from the AP, I had signal strengths of roughly 65-75%. To me that seemed really low given the proximity. I suspected significant interference was causing signal loss. Only when I put the touchpad within say 10 inches of the antenna from the AP did the signal strength go above 90%.

Looking into the large number of receive errors told me that those errors are caused almost entirely by interference.

So then I wanted to see what channels were most being used and try to use a channel that has less congestion, the AP defaulted to channel 6.

The last time I mucked with wifi on linux there seemed to be an endless stream of wireless scanning, cracking, hacking tools. Much to my shock and surprise these days most of those tools haven’t been maintained in 5-6-7-8+ years. There aren’t many left. Sadly enough the default Ubuntu wifi apps do not report channels they just report SSIDs. So I went on a quest to find a tool I could use. I finally came across something called wifi radar, which did the job more or less.

I counted about 25 broadcasting SSIDs using wifi radar, nearly half of them if I recall right were on channel 6. A bunch more on 11 and 1, the other two major channels. My WRT54G had channels going all the way up to 14. I recall reading several years ago about frequency restrictions in different places, but in any case I tried channel 14 (which is banned in the US). Wifi router said it was channel 14, but neither my laptop nor Touchpad would connect. I suspect since they flat out don’t support it. No big deal.

Then I went to channel 13. Laptop immediately connected, Touchpad did not. Channel 13 is banned in many areas, but is allowed in the U.S. if the power level is low.

Next I went to channel 12. Laptop immediately connected again, Touchpad did not. This time I got suspicious of the Touchpad. So I fired up my Palm Pre, which uses an older version of the same operating system. It saw my wifi router on channel 12 no problem. But the Touchpad remained unable to connect even if I manually input the SSID. Channel 12 is also allowed in the U.S. if the power level is low enough.

So I ended up on channel 11. Everything could see everything at that point. I enabled WPA2 encryption, enabled MAC address filtering (yes I know you can spoof MACs pretty easily on wifi, but at the same time I have only 2 devices I’ll ever connect so blah). I don’t have a functional VPN yet mainly because I don’t have a way (yet) to access VPN on the Touchpad, it has built in support for two types of Cisco VPNs but that’s it. I installed OpenVPN on it but I have no way to launch it on demand without being connected to the USB terminal.  I suppose I could just leave it running and in theory it should automatically connect when it finds a network but I haven’t tried that.

So on to my last point on wifi – interference. As I mentioned earlier signal quality was not good even being a few feet away from the access point. I decided to try out speedtest.net to run a basic throughput test on both the Touchpad and the Laptop. All tests were using the same Comcast consumer broadband connection

The test runs in flash, and as you can see of course the Touchpad’s browser (or flash) is not nearly as fast as the laptop, not too unexpected.

Comparing LAN transfer speeds was even more of a joke of course, I didn’t bother involving the Touchpad in this test just the laptop. I used iperf to test throughput(no special options just default settings).

• Wireless – 7.02 Megabits/second (3.189 milliseconds latency)
• Wired – 930 Megabits/second (0.3 milliseconds latency)

What honestly surprised me though was over the WAN, how much slower wifi was on the laptop vs wired connection, it’s almost 1/3rd the performance on the same laptop/browser. I justed measured to be sure – my laptop’s screen (where I believe the antenna is at) is 52 inches from the WRT54G router.

It’s “fast enough” for the Touchpad’s casual browsing, but certainly wouldn’t want to run my home network on it, defeats the purpose of paying for the faster connectivity.

I don’t know how typical these results out there. One place I recently worked at was plagued with wireless problems, performance was soo terrible and unreliable. They upgraded the network and I wasn’t able to maintain a connection for more than two minutes which sucks for SSH. To make matters worse the vast majority of their LAN was in fact wireless, there was very little cable infrastructure in the office. Smart people hooked up switches and stuff for their own tables which made things more usable, though still a far cry from optimal.

In a world where we are getting even more dense populations and technology continues to penetrate driving more deployments of wifi, I suspect interference problems will only get worse.

I’m sure it’s great if the only APs within range are your own, if you live or work at a place that is big enough. But small/medium businesses frequently won’t be so lucky, and if you live in a condo or apartment like me, ouch…

My AP is not capable of operating in the 5Ghz range 802.11a/n, that very well could be significantly less congested. I don’t know if it is accurate or not but wifi radar claims every AP within range of my laptop(47 at the moment) is 802.11g (same as me). My laptop’s specs say it supports 802.11b/g/n, so I’d expect if anyone around me was using N then wifi radar would pick it up, assuming the data being reported by wifi radar is accurate.

Since I am moving in about two weeks I’ll wait till I’m at my new apartment before I think more about the possibility of going to a 802.11n capable device for reduced interference. On that note does any of my 3-4 readers have AP suggestions?

Hopefully my new place will get better 4G wireless coverage as well, I already checked the coverage maps and there are two towers within one mile of me, so it all depends on the apartment itself, how much interference is caused by the building and stuff around it.

I’m happy I have stuck with ethernet for as long as I have at my home, and will continue to use ethernet at home and at work wherever possible.

   Send article as PDF to

oof

What I’m talking about is the announcement of the Black Diamond X-Series from my favorite switching company Extreme Networks. I have been hearing a lot about other switching companies coming out with new next gen 10 GbE and 40GbE switches, more than one using Broadcom chips (which Extreme uses as well), so have been patiently awaiting their announcements.

I don’t have a lot to say so I’ll let the specs do the talking

Extreme Networks Black Diamond X-Series

• 14.5 U
• 20 Tbps switching fabric (up ~4x from previous models)
• 1.2 Tbps fabric per line slot (up ~10x from previous models)
• 2,304 line rate 10GbE ports per rack (5 watts per port) (768 line rate per chassis)
• 576 line rate 40GbE ports per rack (192 line rate per chassis)
• Built in support to switch up to 128,000 virtual machines using their VEPA/ Direct Attach system

This was fascinating to me:

Ultra high scalability is enabled by an industry-leading fabric design with an orthogonal direct mating system between I/O modules and fabric modules, which eliminates the performance bottleneck of pure backplane or midplane designs.

I was expecting their next gen platform to be a mid plane design (like that of the Black Diamond 20808), their previous 10GbE high density Enterprise switch Black Diamond 8800, by contrast was a backplane design (originally released about six years ago). The physical resemblance to the Arista networks chassis switches is remarkable. I would like to see how this direct mating system looks in a diagram of some kind to get a better idea on what this new design is.

Mini RJ21 adapters, 1 plug on the switch, goes to 6x1GbE ports

To put that port density in to some perspective, their older system (Black Diamond 8800), by comparison, has an option to use Mini RJ21 adapters to achieve 768 1GbE ports in a chassis (14U), so an extra inch of space gets you the same number of ports running at 10 times the speed, and line rate (the 768x1GbE is not quite to line rate but still damn fast). It’s the only way to fit so many copper ports in such a small space.

It seems they have phased out the Black Diamond 10808 (I deployed a pair of these several years ago first released 2003), the Black Diamond 12804C (first released about 2007), the Black Diamond 12804R (also released around 2007) and the Black Diamond 20808 (this one is kind of surprising given how recent it was though didn’t have anything approaching this level of performance of course, I think it was released in around 2009). They also finally seemed to drop the really ancient Alpine series (10+ year old technology) as well.

Also they seem to have announced a new high density stackable 10GbE switch the Summit X670, the successor to the X650 which was already an outstanding product offering several features that until recently nobody else in the market was providing.

Extreme Networks Summit X670

• 1U
• 1.28 Tbps switching fabric (roughly double that of the X650)
• 48 x 10Gbps line rate standard (64 x 10Gbps max)
• 4 x 40Gbps line rate (or 16 x 10Gbps)
• Long distance stacking support (up to 40 kilometers)

The X670 from purely a port configuration standpoint looks similar to some of other recently announced products from other companies, like Arista and Force10, both of whom are using the Broadcom Trident+ chipset, I assume Extreme is using the same. These days given so many manufacturers are using the same type of hardware you have to differentiate yourself in the software, which is really what drives me to Extreme more than anything else, their Linux-based easy-to-use Extremeware XOS operating system.

Neither of these products appear to be shipping, not sure when they might ship, maybe sometime in Q3 or something.

40GbE has taken longer than I expected to finalize, they were one of the first to demonstrate 40GbE at Interop Las Vegas last year, but the parts have yet to ship (or if they have the web site is not updated).

For the most part, the number of companies that are able to drive even 10% of the performance of these new lines of networking products is really tiny. But the peace of mind that comes with everything being line rate, really is worth something !

x86 or ASIC? I’m sure performance boosts like the ones offered here pretty much guarantees that x86 (or any general purpose CPU for that matter) will not be driving high speed networking for a very long time to come.

Myself I am not yet sold on this emerging trend in the networking industry that is trying to drive everything to be massive layer 2 domains. I still love me some ESRP! I think part of it has to do with selling the public on getting rid of STP. I haven’t used STP in 7+ years so not using any form of STP is nothing new for me!

   Send article as PDF to

But this time it is AT&T imposing caps on their various broadband plans. I don’t know whether to laugh at or feel sorry for some of these people (see the comments on the site) that believe they have a right to maximum performance, unlimited bandwidth for a few bucks a month.

**** YOU and your troll crap. DONT BE MAD BECAUSE IM TELLING THE TRUTH YOU AT&T DRONE. YOU WEEP FOR THE COMPANY WHO HAS MORE MONEY THAN THE U.S TREASURY. GET YOUR HEAD EXAMINED.

IF YOUR CRAPPY DSL IS SLOW IT’S BECAUSE YOUR ISP IS TOO CHEAP TO UPGRADE TO THE NEEDS OF THE WORLD IN 2011!!!!

This post really is funny

If only two percent of people are affected, why do you feel the need to screw the rest of the 98%?!

As a Comcast broadband customer I have a 250GB cap a month. I have no doubt though that I fall far short of the cap, I’d be surprised if I do more than 20GB a month (that is with occasional netflix streaming though these days I can’t find anything I want to stream on Netflix, I’ve watched one or two things in the past month) UPDATE – I forgot Comcast does have a bandwidth meter you can check, so I got my account info and checked it out. I wonder where I stand as far as a percentile of their customers – low usage on average? medium?

I do run a server as well in the Terremark cloud, so I checked out the bandwidth on it as well (which hosts this blog along with my email services, other web sites etc)

• February – 6GB data transfer (I assume they charge on inbound and outbound transfers?)
• January – 2GB data transfer
• December – 2GB data transfer

The world is built in over subscription, that’s a big driver to keeping costs low. Whether it’s bandwidth, or phone/mobile call capacity, or even your local grocery store.

I for one think AT&T’s plan is very reasonable, they will charge you $10 per 50GB over their limits, $100 for 500GB of data transferred. They will also provide notifications when you hit certain levels of that cap.

The big mistake all of these providers made was of course to offer unlimited plans in the first place.

   Send article as PDF to

He discussed what Stifel’s report calls “continued lacklustre FCoE adoption.” FCoE is the running of Fibre Channel storage networking block-access protocol over Ethernet instead of using physical Fibre Channel cabling and switchgear. It has been, is being, assumed that this transition to Ethernet would happen, admittedly taking several years, because Ethernet is cheap, steamrolls all networking opposition, and is being upgraded to provide the reliable speed and lossless transmission required by Fibre Channel-using devices.

Maybe it’s just something specific to investors, I was at a conference for Brocade products I think it was in 2009 even, where they talked about FCoE among many other things and if memory serves they didn’t expect much out of FCoE for several years so maybe it was management higher up that was setting the wrong expectations or something I don’t know.

Then more recently I saw this article posted from slashdot which basically talks about the same thing.

Even today I am not sold of FCoE, I do like Fibre Channel as a protocol but don’t see a big advantage at this point to running it over native Ethernet. These days people seem to be consolidating on fewer, larger systems, I would expect the people more serious about consolidation are using quad socket systems, and much much larger memory configurations (hundreds of gigs). You can power that quad socket system with hundreds of gigs of memory with a single dual port 8Gbps fibre channel HBA.Those that know about storage and random I/O understand more than anyone how much I/O it would really take to max out an 8Gbps Fibre channel card, your not likely to ever really manage to do it with a virtualization workload, even with most database workloads. And if you do you’re probably running at a 1:1 ratio of storage arrays to servers.

The cost of the Fibre network is trivial at that point (assuming you have more than one server). I really like the latest HP blades because well you just get a ton of bandwidth options with them right out of the box, why stop with running everything over a measly single dual port 10Gbe NIC when you can have double the NICs, AND throw in a dual port Fibre adapter for not much more cash. Not only does this give more bandwidth, but more flexibility and traffic isolation as well(storage/network etc). On the blades at least it seems you can go even beyond that(more 10gig ports), I was reading in one of the spec sheets for the PCIe 10GbE cards that on the Proliant servers no more than two adapters are supported

NOTE: No more than two 10GbE I/O devices are supported in a single ProLiant server.

I suspect that NOTE may be out of date with the more recent Proliant systems that have been introduced, after all they are shipping a quad socket Intel Proliant blade with three dual port 10GbE devices on it from the get go. And I can’t help but think the beast DL980 has enough PCI busses to handle a handful of 10GbE ports. The 10GbE flexfabric cards list the BL685c G7 as supported as well, meaning you can get at least six ports on that blade as well. So who knows…..

Do the math, the added cost of a dedicated fibre channel network really is nothing. Now if you happen to go out and chose the most complicated to manage fibre channel infrastructure along with the most complicated fibre channel storage array(s) then all bets are off. But just because there are really complicated things out there doesn’t mean your forced to use them of course.

Another factor is staff I guess, if you have monkeys running your IT department maybe Fibre channel is not a good thing and you should stick to something like NFS, and you can secure your network by routing all of your VLANs through your firewall while your at it, because you know your firewall can keep up with your line rate gigabit switches right? riiight.

I’m not saying FCoE is dead, I think it’ll get here eventually, I’m not holding my breath for it though, it’s really more of a step back than a step forwards with present technology.

   Send article as PDF to