10 Gigabit Ethernet has been around for many years, for much of that time it has been for the most part(and with most vendors still is) restricted to more expensive chassis switches. For most of these switches the port density available for 10GbE is quite low as well, often maxing out at less than 10 ports per slot.
Within the past year Extreme Networks launched their X650 series of 1U switches, which currently consists of 3 models:
- 24-port 10GbE SFP+
- 24-port 10GbaseT first generation
- 24-port 10GbaseT second generation (added link to press release, I didn’t even know they announced the product yesterday it’s been available for a little while at least)
For those that aren’t into networking too much, 10GbaseT is an ethernet standard that provides 10 Gigabit speeds over standard CAT5e/CAT6/CAT6a cable.
All three of them are line rate, full layer 3 capable, and even have high speed stacking(ranging from 40Gbps to 512Gbps depending on configuration). Really nobody else in the industry has this ability at this time at least among:
- Brocade (Foundry Networks) – Layer 2 only (L3 coming at some point via software update), no stacking, no 10GbaseT
- Force10 Networks – Layer 2 only, no stacking, no 10GbaseT
- Juniper Networks – Layer 2 only, no stacking, no 10GbaseT. An interesting tidbit here is the Juniper 1U 10GbE switch is an OEM’d product, does not run their “JunOS” operating system, and will never have Layer 3 support. They will at some point I’m sure have a proper 10GbE switch but they don’t at the moment.
- Arista Networks – Partial Layer 3(more coming in software update at some point), no stacking, they do have 10GbaseT and offer a 48-port version of the switch.
- Brocade 8000 – Layer 2 only, no stacking, no 10GbaseT (This is a FCoE switch but you can run 10GbE on it as well)
- Cisco Nexus 5000 – Layer 2 only, no stacking, no 10GbaseT (This is a FCoE switch but you can run 10GbE on it as well)
- Fulcrum Micro Monte Carlo – I had not heard of these guys until 30 seconds ago, found them just now. I’m not sure if this is a real product, it says reference design, I think you can get it but it seems targeted at OEMs rather than end users. Perhaps this is what Juniper OEMs for their stuff(The Fulcrum Monaco looks the same as the Juniper switch). Anyways they do have 10GbaseT, no mention of Layer 3 that I can find beyond basic IP routing, no stacking. Probably not something you want to use in your data center directlty due to it’s reference design intentions.
The biggest complaints against 10GbaseT have been that it was late to market(first switches appeared somewhat recently), and it is more power hungry. Well fortunately for it the adoption rate of 10GbE has been pretty lackluster over the past few years with few deployments outside of really high end networks because the cost was too prohibitive.
As for the power usage, the earlier 10GbaseT switches did use more power because well it usually requires more power to drive stuff over copper vs fiber. But the second generation X650-24T from Extreme has lowered the power requirements by ~30%(reduction of 200W per switch), making it draw less power than the SFP+ version of the product! All models have an expansion slot on the rear for stacking and additional 10GbE ports. For example if you wanted all copper ports on the front but needed a few optical, you could get an expansion module for the back that provides 8x 10GbE SFP+ ports on the rear. Standard it comes with a module that has 4x1GbE SFP ports and 40Gbps stacking ports.
So what does it really cost? I poked around some sites trying to find some of the “better” fully layer 3 1U switches out there from various vendors to show how cost effective 10GbE can be, at least on a per-gigabit basis it is cheaper than 1GbE is today. This is street pricing, not list pricing, and not “back room” discount pricing. YMMV
| Vendor | Model | Number of ports on the front | Bandwidth for front ports (Full Duplex) | Priced From | Street Price | Cost per Gigabit | Support Costs? |
|---|---|---|---|---|---|---|---|
| Extreme Networks | X650-24t | 24 x 10GbE | 480 Gbps | CDW | $19,755 * | $41.16 | Yes |
| Force10 Networks | S50N | 48 x 1GbE | 96 Gbps | Insight | $5,078 | $52.90 | Yes |
| Extreme Networks | X450e-48p | 48 x 1GbE | 96 Gbps | Dell | $5,479 | $57.07 | Optional |
| Extreme Networks | X450a-48t | 48 x 1GbE | 96 Gbps | Dell | $6,210 | $64.69 | Yes |
| Juniper Networks | EX4200 | 48 x 1GbE | 96 Gbps | CDW | $8,323 | $86.69 | Yes |
| Brocade (Foundry Networks) | NetIron CES 2048C | 48 x 1GbE | 96 Gbps | Pending | Pending | Pending | Yes |
| Cisco Systems | 3750E-48TD | 48 x 1GbE | 96 Gbps | CDW | $13,500 | $140.63 | Yes |
* The Extreme X650 switch by default does not include a power supply(it has two internal power supply bays for AC or DC PSUs). So the price includes the cost of a single AC power supply.
How do you feel about the viability of Extreme Networks as a company. I read that thaey had some pretty massive layoffs (who hasn’t) and a new CEO. I was considering using them as a core switch for some high end vSphere servers to plug into and use something more traditional like Cisco on the access layer. The reason I was considering this is because of the competitive pricing for 2nd gen 10Gb Ethernet. I’m still concerned about interop though and I’d like my virtualized environment to be as aware as possible of the switching infrastructure and vice versa (something like the integration you see with the Nexus 1000v and the hardware switching infrastructure).
Thanks for any feedback and by the way this is Scott from the CentOS user list (thanks for the storage discussion awhile back)!
Comment by Scott — November 17, 2009 @ 9:30 pm
I’ve had to deal with this type of question for years now since they are a smaller player. I look at it like this, they’ve been making products for 13 years now, they have a solid customer base with more than 20 million ethernet ports installed(as of some time last year anyways, that was an estimate by some 3rd party). They went through some restructuring recently to position the company to be profitable at a lower revenue numbers. Which to me is a good thing, that means they understand the economic environment and are adjusting to the “new normal”. They have had a slew of new product announcements and launches in the past year and with their recent changes to include lifetime warranties on several products(X450e, Black Diamond 8500, X350 etc) I think that will bode very well for them going forward, as it gives them a good advantage over HP(who has lifetime warranty on most/all ProCurve products, but the ProCurve line lacks the capabilities of the Extreme stuff, so now you can get the best of both worlds).
If you have Extreme reps in your area I’d suggest talking to them they have an interesting virtualization strategy, the guy who invented or created(or something) the recent Cisco virtualization stuff is now working for Extreme. And I’m told that VMware themselves use Extreme almost exclusively in house. Now that they are partnered closer with Cisco that might change, but that’d be more for political reasons than anything technical.
And even if you don’t like Extreme for whatever reason(their OS is very different,their command structure is much easier to learn, but if you have a strong background in IOS it may be a shock to your mind that it’s such a logical way of doing things, I have another blog entry comparing Extreme to Juniper commands you can look up), there are other solid vendors out there as well, I view Foundry as sort of like Cisco just with better hardware, and much lower cost. Juniper is still lacking in the switching space(of course everyone knows they make good routers, but I don’t run a service provider). Force10 is good I’m told as well.
As one of my boss’ friends told him when we were having this very discussion at my company, with Cisco you’ll pay twice as much and not get a better product. If you like the IOS command structure then look to Foundry, or Force10(which I’m told is very Cisco-like as well). If your open to something different, check out Extreme.
Glad to see there is another reader of the blog! thanks for reading.
(Hmm strange that the blog didn’t format my comment right, I put in several line breaks, while they show up in the editor box they don’t show up in the finished product.)
[edit – removed comment about removing access layer I meant aggregation layer]
Comment by Nate — November 17, 2009 @ 10:09 pm
I think you should take another look at Arista. If you had put their 7120T-4S and 7140T-8S products in your table I think they would look attractive by comparison to the other products, with a price per gigabit around $31.
I am curious about Arista’s layer 3 capabilities, though. What have you heard about it that is “very limited” in comparison to the Extreme layer 3 support?
Also, I don’t see the Cisco Nexus 5000 products on your list — why not? I wonder if Cisco is going to come out with a 10GBASE-T version of the Nexus 5000?
Comment by Anonymous — November 17, 2009 @ 10:27 pm
Cisco Nexus is not really an apples to apples comparison as it is a “converged” switch. I think the closest comparison would be the Brocade 8000. The cost of a converged switch is quite a bit more, since it needs to do different things.
As for Arista’s layer 3 support, I don’t have personal experience with their products but their own data sheet says VRRP, BGP and IPv6 routing isn’t available yet. I don’t need BGP or IPv6 but VRRP is pretty critical in a layer 3 enivornment. Myself I prefer ESRP(Extreme Standby Router Protocol) over VRRP because it’s much simpler to operate, I still plan to write a blog entry on that as well soon.
Extreme doesn’t tout ESRP enough, I’m pushing them to pay more attention to it. They do push EAPS which is a layer 2 ring protocol with sub 50ms recovery times. Some other vendors(Foundry, with MRP and I think Force10 with their FRRP) have similar protocols. But they need VRRP layered on top for layer 3. While ESRP provides layer 2 and layer 3 protection in one protocol and outside of the core network(usually two switches) works with any switch vendor downstream.
If nothing else I’d want Extreme at the core of any switching network I run for ESRP alone, it’s that great of a protocol. Wish other companies would come up with something to do the same on their gear but to my knowledge nobody has.
In addition on the data sheet Arista doesn’t target their switch to perform routing as you can see in the “Target Positioning” section. This will probably change in the future though.
As for pricing that is interesting, I will be keeping my eye on them. I was told recently Arista lost out to the X650 here at one deal locally partly due to the X650 being cheaper, though that may of been because of back room pricing rather than street pricing, I don’t know. $31/gbit is quite good though.
I hope that they can broaden their portfolio of products as well. We’re using 1GbE at our customer facing edge sites, and at least for the moment the main plans for 10GbE is at the core, mainly for uplinking edge switches to the core, rather than using 1GbE trunks. The only 10GbE systems will be blades running VMware using HP VirtualConnect technology, and our NAS cluster which I hope to upgrade to 10GbE next year as well(right now it uses many 1GbE ports).
Does Arista support sFlow? That is another key component to a high speed network these days. In fact it was what knocked Force10 out of the running for our most recent decision at my company, their 1GbE switch doesn’t have sFlow(I’m told it will get it in the future, don’t know if it’s a software only change or if it needs a hardware refresh as sFlow is usually integrated into the hardware).
Extreme supports sFlow, and they have another product called ClearFLOW which while sounds similar is very different in that it is sort of like a NIDS that is integrated into the switching fabric. I’ll write another entry on that at some point too.
Wow such great comments! thanks !!
Comment by Nate — November 17, 2009 @ 11:04 pm
Oh and if you haven’t already check out my comments on FCoE, I wrote an entry on it a few months ago. It kind of addresses the point as to comparing the Nexus 5000 and the Brocade 8000 to normal ethernet switches, the whole DCE movement(if that’s what it’s called this week?).
http://www.techopsguys.com/2009/08/17/fcoe-hype/
Comment by Nate — November 17, 2009 @ 11:08 pm
I find it interesting that you are critical of the Nexus 5000 because of its FCoE features. You don’t need to use those features, and the Nexus 5000 is a decent plain vanilla 10GbE Ethernet switch, although only L2. Most of the FCoE features require a separate software license, which is expensive, but the base Nexus 5010 and 5020 switches are not outrageously priced, especially once you factor in Cisco’s massive discounting from list price. Anyone who pays more than 40% of Cisco’s list price is being abused. The actual selling price of a Nexus 5010 (without transceivers, without add-in cards, without all the crazy extra software licenses) is less than $10K. As long as you can convince your Cisco sales rep that you are leaning toward purchasing from another vendor and that price is the main criteria you can get them to slash prices. Not having purchased any Extreme gear in many years I don’t know what their current discounting strategy is, but show your Cisco rep a quote from Arista Networks and see them scramble to win the deal!
Arista Networks doesn’t list sFlow on their datasheet, so probably they don’t support it (yet?). The fact that those L3 features are listed with little asterisks next to them (BGP, VRRP, IPv6) on the data sheet probably means they are going to be available soon — if you are in the market for a switch comparable to what they sell I’d contact them and see about availability of a software update that includes what you need. VRRP should be easy for them to deliver.
As far as broadening their portfolio, I saw that Arista now has a 48 port 1GbE switch with 4 10GbE uplinks.
Comment by Anonymous — December 2, 2009 @ 10:39 pm
Yes I’ve been watching Arista for a year or two now, noticed their 48 port 10GbaseT switch a while back.
I’ll add in the Cisco Nexus and Brocade 8000 soon, though as you mentioned layer 2 only, and of course no stacking, and no 10GbaseT. For Cisco 20-port street price of $23,400 (including 1 PSU and 20 short range SFP+ modules) doesn’t sound like a good deal to me! Fewer (ethernet)features, fewer ports, for more $.
If you haven’t already and are curious my thoughts on FCoE are here http://www.techopsguys.com/2009/08/17/fcoe-hype/
Comment by Nate — December 3, 2009 @ 9:17 am
I think your street price on the Nexus 5010 is higher than what any customer who buys more than one is going to pay. Oh, I see you are including SR SFP+ modules, and since Cisco charges indecent prices for those, it drives the quote WAY up. Breaking that $23.4K down the SR modules must be half the quote, right?
Typical deployment for the Nexus 5010 is either 10G server attach via SFP+ direct attach twinax copper cables, or 1G server attach via the Nexus 2000 port expander, or a mix of both.
Re: my Arista “portofolio broadening” comment: I was not referring to their 7140T-8S (40 port 10GBASE-T, 8 port SFP+) box. I was referring to their 7048T 48 port 1000BASE-T/4 SFP+ box. I mentioned this because you expressed interest in a 1G solution. I’ll quote you:
“We’re using 1GbE at our customer facing edge sites, and at least for the moment the main plans for 10GbE is at the core, mainly for uplinking edge switches to the core, rather than using 1GbE trunks”.
The Arista 7048T seems like a good fit for that type of situation. It has crazy amounts of packet buffer so you won’t drop packets if you get bursts from the 10G side that are destined to 1G ports. I had a crazy bad experience with some Foundry FastIron 1G switches that would drop packets like a blind juggler. There was a storage server connected to one of the 10G uplink ports and the FastIron only had like 100KB of buffer per port. That’s okay as long when no 10G-to-1G speed conversion is happening, but it SUCKS when you have fast servers on 10G links in the network that can blast traffic down to the 1G servers.
If Extreme’s X650 is beating Arista in deals because of pricing Extreme must be giving some really interesting discounts since the list prices for comparable products don’t look competitive. How do I get hooked up so I can put pressure on my other vendors (Can you say “stalking horse”… hehhehehe)? Can you post some example pricing? Is the situation you referenced the 10GBASE-T version of the X650, or the SFP+ version, and were they comparing apples to apples with the equivalent Arista switch?
Comment by Anonymous — December 6, 2009 @ 10:25 pm
I tried poking around for online street pricing for Arista stuff but came up empty. The Cisco pricing I got from a few different places, certainly not big discount pricing just online web pricing(same style pricing I got for everyone else). I included the costs of the SFP+ modules because you have to have them for the ports to work. Not much I can do if the vendor doesn’t offer 10GbaseT which lowers the cost of course(the point of the post!)
It is pretty crazy the amount of discounts the vendors are giving these days, whether it’s in networking and storage and probably a few other areas too. As for how you get a hookup, at this point I think it’s best if you have a good VAR, there are a lot of them out there, most of them aren’t very good. My own (newly appointed) VAR was trying to push Force10, then push Juniper, I got him to sell Extreme now too(his own network engineers who are not local love Extreme he just wasn’t aware of them too much). A good VAR that sells a lot of gear can have quite a bit of pull, more so than the customer(unless the deal is really big), so they can often get better pricing than the customer can going direct(even adding in the margins the VAR gets).
But going direct can help too, it depends on the regional rep. My current rep isn’t the rep I started out with 5 years ago, but the company knows me so I do probably get some special treatment because of the long running relationship.
I don’t know if the X650 is beating the Arista everywhere(I’m sure it’s not), I just heard of a couple places. Arista certainly has a unique product in their 1U 40+ port offerings that nobody else can match so it depends on the requirements.
For me as my posts since this one have tried to illustrate it’s the software as much as the hardware. Arista’s virtual EOS looks interesting to be sure, though lacking mature full layer 3 for me is a tough sell(if I had a really massive networks and used their gear in a small portion of it, it wouldn’t be as big of a deal, my networks are smaller though).
Comment by Nate — December 7, 2009 @ 8:07 am
My objection to the Nexus 5010 pricing was that you do *not* need SR SFP+ modules in a typical deployment because you use SFP+ direct attach copper cables instead. Cisco calls this “10GBASE-CU” I believe. If you’re not familiar with these, they are good up to 5 meters or so and cost <$100 per port with typical Cisco 50% discounting, more like $60 per port for the shorter lengths (1m, 3m). They are cheaper, use less power, and no need for a fiber patch cord either.
Comment by Anonymous — December 7, 2009 @ 9:53 am
ahh ok, I admit I am far from fluent in the cisco part numbers so just picked something that looked right..
Comment by Nate — December 7, 2009 @ 10:06 am
You are partially right about your speculation about where Juniper gets their EX2500 switch: it does contain Fulcrum silicon, but it is a rebranded BNT G8124, not a Fulcrum reference design.
Comment by Anonymous — December 12, 2009 @ 2:27 am
That’s nice to know about 10 Gigabit Ethernet.
Comment by 1000base sfp — May 24, 2010 @ 9:28 pm
I’ve seen this Arista pricing, and it shouldn’t be too hard to get.
~$13,500 for the Arista 7120T
~$22,500 for the Arista 7140T
Comment by ATLDude — May 26, 2010 @ 10:04 am
I forgot to mention, Brocade has the Turboiron 24x. It’s has 24 10/1GBE ports for ~$8000.
Comment by ATLDude — May 26, 2010 @ 10:32 am
[…] have blogged on some of my more favorite topics in the past, with regards to their technology. I’ve been using Extreme stuff for just about […]
Pingback by Moving on up to Number two « TechOpsGuys.com — August 25, 2010 @ 4:12 pm
[…] it’s true… I’ve been a fan of 10GbaseT for a while now. Though it hasn’t really caught on in the industry, off the top of my head I can only think […]
Pingback by 10GbaseT making a comeback ? « TechOpsGuys.com — March 19, 2012 @ 12:20 pm