Diggin' technology every day

November 13, 2012

100GbE: Still a very hefty premium

Filed under: Networking — Tags: , — Nate @ 11:22 am


Big Switch Networks decloaked today, and released their new OpenFlow controller, in partnership with many different networking vendors.

Arista Networks, Dell, Brocade, Juniper Networks, Brocade Communications, and Extreme Networks have all partnered with Big Switch, and their OpenFlow-enabled switches are certified to be control-freaked by Big Network Controller. Switches from IBM and HP have been tested for interoperability, but there are no formal partnerships.

All of this SDN stuff really is sort of confusing to me (it really seems like the whole software defined thing is riding on a big hype cloud). One thing that stands out to me here is that this OpenFlow stuff seems to only cover switching and routing. I don’t see any mention of things like firewalls, or more importantly – load balancers.  Maybe those folks will integrate with OpenFlow at some point in some way.

On this article A10 Networks (load balancing company) is mentioned as a partner, but running a search for either OpenFlow or BigSwitch on the A10 site reveals no results.

For me if I’m going to be moving workloads between datacenters, at least those that deal with internet connectivity, I certainly want that inbound connectivity to move to the new datacenter as well, and not incur the costs/latency of forwarding such traffic over a back end connection. The only exception being if there is a fault at the new datacenter which is severe enough to want to route internet traffic from another facility to it. I suppose at the same time the fault would likely have to block the ability of moving the workload to another (non faulty) facility.

F5 networks had a demo they put out on long distance vMotion almost three years ago. Using their WAN Optimization, their Global Traffic Managers(Global DNS), and Local Traffic managers(load balancers), it was a pretty cool setup. Of course this was ages before VMware had such a solution in house, and I believe this solution (for the niche that it serves) can cover a significantly longer distance than what you get with VMware today.

Anyway that’s not the topic of the post. At the same time I noticed Extreme announced their first 100GbE offering (per usual it looks like it won’t be available to ship for at least 6 months – they like to announce early for some strange reason). On their X-8 platform which has 1.2Tbps of throughput per line card, and up to 20Tbps (15Tbps non blocking even with a fabric failure) per chassis. I say “up to” because there are multiple fabric modules, and there are two different speeds(2.5Tbps and 5Tbps).

The card is a combo 4-port 100GbE card. They also announced a newer larger scale 12-port 40GbE line card. What struck me(still) was the cost distinction between the two:

NTE list pricing includes: 40GbE 12 port XL module at US $6,000.00 per port; 100GbE 4 port XL module at US $35,000 per port.

I think I recall hearing/reading last year that 100GbE was going for around $100,000/port, if so this would be a great discount, but still pretty crazy expensive compared to 40GbE obviously!

UPDATE – It seems my comment was lost in the spam, the lack of approval wasn’t intentional.

While I’m here let me rag on Extreme a bit here – I posted a comment on one of their blog posts (about 3 weeks ago) where they said they moved away from designing their own ASICs with the X-8 platform.

They never approved the comment.

My comment was basically asking them when their last ASIC design was – to my knowledge their last ASIC was the 4GNSS ASIC (they called it a programmable ASIC – I assume that meant more of a FPGA but who knows), that originally showed up in the Black Diamond 10808 back in 2003(I had a pair of these boxes in 2005). I believe they re-used it, perhaps refined it a bit in the following years but don’t believe any new ASICs were designed since (sure I could be wrong but they haven’t clarified). So I’d say their last ASIC design was more than a decade ago, and only now this blogger comes out and says they don’t do ASICs any more. Before that the last one I know of was their Inferno chipset, a much better name, which was present in their older platforms running on the original ExtremeWare operating system, the last such switches to be sold were in their Alpine series and the Summit 48si (I still have one of these at home but it doesn’t do much today – too loud for home use).

Anyway, shame on you for not approving my reasonable response to your post!

btw I approve all posts here, even those that try to attack me/my posts. If for some reason your post is not immediately available, contact me (see blurb on right) because your post may of been caught by the SPAM filter. I don’t go through those caught posts often(there are a lot), maybe 2-3 times a year.


  1. Nate, sorry about your comment, which I have gone back and approved. Legit comment that was lost in a sea of blog comment spam, spam which in fact prompted us to upgrade the CAPTCHA widget in blog.

    Comment by Jason Lackey — November 13, 2012 @ 11:29 am

  2. ok thanks! didn’t know you read my blog too 🙂

    I know how the sea of comment spam feels.

    Comment by Nate — November 13, 2012 @ 11:49 am

  3. Nate,
    You should see if you can add a simple spam filter. I don’t like captchas myself. Instead, for my WordPress site, I use TanTanNoodles Simple Spam filter (which blocks comments with more than a set number of links) + Akismet. The Simple Spam filter really helps by getting rid of most of the robo-spam; I’d say it reduced my spam from >10/day to <1 per day.

    Comment by Tony — November 13, 2012 @ 12:53 pm

  4. Hey Tony –

    Yes I do have a spam filter (Akismet) – more than 35k comments blocked by it according to the graphic on the right side. About 1 spam gets through Akismet every couple of months – is there an advantage to something like TanTan on top of Akismet ?

    I am by no means a word press expert, Akismet was setup by someone else years ago and I just moved it to the new servers as I moved things around since it seemed to work OK.


    Comment by Nate — November 13, 2012 @ 1:06 pm

  5. I was also interested in OpenFlow as a way to implement L4 load-balancing on the cheap using commodity switches. There is a project to do so, but I don’t think OpenFlow has all the functionality needed yet for a complete solution:

    Another cool application would be for companies that do NAT traversal bridging in the cloud, e.g. STUN/TURN. The initial setup would be done on a server but everything afterwards handled in switch fabric ASICs.

    Comment by Fazal Majid — November 13, 2012 @ 2:28 pm

  6. Hi Nate,
    If you want to occasionally check Akismet for false positives (real comments that Akismet thinks are spam), then the Simple Spam Filter is worth checking out. I was going through my Akismet spam about weekly to check if something legit got tagged as spam, and didn’t enjoy it; I like it a lot better now with the added filter, since so much less spam makes it to Akismet. But if you never check Akismet’s spam, adding the extra filter isn’t worth it.

    Comment by Tony — November 13, 2012 @ 2:46 pm

  7. That looks interesting thanks for the link. I read through their presentation PDF. It looks like the main purpose of the project is to load balance multiple IDSs to direct flows to, mainly running snort. In their analysis I did not see them talk at all about integrating something like sFlow or Netflow into an IDS and using that instead of relying on very heavy usage of port mirroring, which regardless of load balancing or not is really bandwidth intensive, and not really scalable on large networks at least.

    Their own tests showed something like 3Gb/s of throughput and upwards of 3-10% or more of lost packets. Can’t imagine what it might look like at 10Gbps or even 40Gbps..

    Extreme came out with a product called ClearFLOW back in 2003/2004. The concept of it was the switch itself would perform basic analysis on the data going over the switch(at line rate), and then make a determination whether or not something needed further analysis(in some cases could act on it immediately), if so then it could mirror the traffic (or send sFlow packets) to an IDS which could look at it in more detail, and then inform the switch whether or not to do anything with the traffic. With emphasis on doing as much within the network fabric as possible though, since it is faster.

    It does require a decent amount of CPU horsepower on the switch depending on the number and complexity of the rules (the original chassis switch offering it could go to 128,000 rules). Unfortunately the technology did not seem to take off, it’s still available in most of their products but not talked about anymore. I suppose it was ahead of it’s time.

    Comment by Nate — November 13, 2012 @ 2:52 pm

  8. I have not tried it, but I believe CloudFare on top of having a very good CDS, also have anti comment spam options. Those are IIRC above the free level, from memory, but modest money, and I am increasingly looking whenever I load a site that is genuinely fast, if they use CloudFare, and turning up positives. I’m not even a customer yet, and not affiliated with them in the slightest, but the thing that has held me back is their system relies on running their DNS. That I am not yet sure about, especially as I never encountered CloudFare before I switched to UltraDNS. I am super careful when it comes to who touches my MX records in business.

    Nate, I must get back into my project with OpenFlow and virtual routers and switching. But ISDs were mentioned, and without having the resources to really test any lately, I hear Sophos can beat out IronPoint. The thing is that Sophos allow for personal use their tech in VMs, and I wonder if that might be helpful for you? I think the sole feature difference si they do not offer content protection and DRM things (corporate ensuring flagged files do not pass the boundary), but that shouldn’t be a hassle for a blog. Maybe worth a look?

    If I can get up to speed with OpenFlow, I am biting the bullet with that. I am forgoing traditional iron entirely, unless something falls down. Possibly two of the sites could receive real amounts of traffic. (One will be concerning a famous reference work that is highly controversial. Schedules are really not fixed just yet, but they will happen soon enough. I have not found much on-line discussion about OpenFlow, let alone getting it to work and play nice with a bunch of other systems. My next step was delayed: running the order for enough hardware to have a modest in house server farm to play with this cloudy stuff. But if that pans out, I reckon we might be saving initially really enough to take the risks. It’s not like I am migrating revenue critical systems, instead building them. Oh, I’d demur and avoid the blame risk and but Extreme kit and the name people on a transaction system. But I am almost past worrying that software router and switch will blow me down on non transactional front of house stuff. If I get both up and running, _and have enough time, might make a good blow by blow account. Some of this is still going through legal, but the name of the origin of one of these sites and why it exists is fairly high PR profile. My daily job now is working out what we can implement for real (this did start a good while ago) which will run clean and make sense to who looks at it.

    May I please rejoin on the sysadmin as a coder Q?

    Think you mentioned earlier in another post, how AWS forces way deeper than simple scripting to get the advantage of it. Reluctantly, I forced a all stop on evaluating AWS about mid summer, for that reason. ONly so many brains here, and that seemed like a zero sum game, at best. Trading our time for hoped for savings or facility. I’m sure that works for “success stories” where the talent is young, time feels so cheap because life seems so long. But that won’t cut it for us. (The fun of OpenFlow is it almost allows us to get what we want out of cloudy ideas, but with our own hardware) … anyhow, I think it is not productive to get talented admin guys delving deep into network programming, for example, use their talent to abstract and plan and see a higher view, for sure. But I meant in my earlier comment on the other thread, that I want admin guys to learn general purpose programming, not merely to dive deeper into their realm. I think it so so much more valuable to get people to move up the food chain, and then you can go hire someone to fill a person’s shoes. That way the incoming admin immediately reports to a sympathetic person, knows that they have a possible career advancement, and generally is encouraged to pay more (but not necessarily more in terms of sweating hours) attention. In a ideal world, I’d have some kind of hypercube where everyone brought someone up in training or tuition to their new position, and that all links back, so that the contact surface of our company for potential hires was all the better. When I think about the business arguments for “upgrading” a great admin to a bloody good programmer, they make me feel extremely positive.

    all best from me ~ j

    Comment by John (other John) — November 17, 2012 @ 7:47 pm

  9. Just to note, I think what Fazal might want to do with OpenFlow is maybe more “enterprizey” than what I need. My project is clean slate, and I’m not going to run existing critical systems over OpenFlow. What I am doing, is risking revenue facing systems to use OpenFlow, because they are nascent, and I estimate the restrictions or possible uncertainty of Open Flow are not significant in the business model.

    For a atto second being on topic, I had not realised that 100Gb/s was into production already! I have felt for too long we are spoiled with developments. Not always, mind. It really has accelerated to puke up ever more amazing possibilities in just this last decade. I often feel intimidated just a bit, because when I do my best work, I kind of force a mental relapse into the 80s me, so that I can concentrate on the simplest ideas or impolementation. Oh, I really must try to find out how to connect and write about how I think so much 80s nostalgia is to do with the refined simplicity of high tech of the era, particularly with consumer electronics, because in one way everything just had so much attention, and there was far less of everything. Have a look see how the numbers of dials and buttons on a camera (counting controls would really change the magnitude) as film departed and digital came in. 33 on my last pro Nikon. Just on the read of the body! Now, with this gluttony we may indulge, where everything but the worse setup and code, or a Death Wish to destroy all functionality, basically works at least in some way, well, I think we are spoiled to the point we do not bother. One thing I saw recently, because I wanted to work on some new styling for our company letters which had started to date, well, when I last looked, the handful of typography and design blogs were all of spectacular quality. Now the blogosphere is full of such utter banality, and almost every other week HN has a discussion where a kindergarten level of design comprehension is lionised and discussed in faux erudition, well, I want to give up on this life. Okay, give up reading the intarwebs, anyhow!

    Narrowly avoiding a rant, over and out from me! ~ j

    Comment by John (other John) — November 17, 2012 @ 8:04 pm

  10. I paid/donated a bit to Akismet, I’m certainly not adverse to paying for something good, I’m not sure if I’ll implement another layer of spam protection, I’m hoping the notice I put up on my site on the right side is enough – if your comment is not posted let me know and I’ll find it and post it. I’ve yet to have someone request that of me. The only person I know that has trouble posting is Calvin from HP Storage blogs. All of his posts had been marked as spam. Though the content of them was always fine, no idea why it was marked that way.

    I’ve liked Sophos for a while, well originally going back to my first exposure to them in 2001 they were great. I haven’t used them much since since I haven’t been in internal IT. My one complaint about Sophos is there has never been a way (to my knowledge) for end users to buy their stuff. You can only get it as a corporate customer. They of course allow employees free license to use at home, but if your employer isn’t using it then you can’t get it (I’ve tried on several occasions).

    Per OpenFlow – what exactly are you wanting to accomplish there? Is it mostly an experiment to learn how it works or is there something it can do that you need that can’t be done via other methods. To me using OpenFlow just makes the network more complex. Complexity can cause more problems than it’s worth in a lot of cases in my experience.

    Interesting to see your experience on AWS mirrors mine and other’s. Glad I’m not alone in that argument 🙂 Sadly many non tech (even if they are in tech roles) managers don’t understand/care about that fact and still force it’s use on the employees.

    thanks again for the comments!!

    Comment by Nate — November 20, 2012 @ 9:04 am

  11. Nate,

    for personal customers, Sophos let you download ISOs of their good stuff.

    For buying it for real in earnest, well I remember a very lovely lady leading my way to that.

    Such a rare example of a Limey firm being both good and charming, they get too little recognition.

    ~ j

    Comment by John (other John) — December 6, 2012 @ 11:35 pm

  12. p,s, may be about to put mission critical voice (international and not simple) onto AWS, though with a specialist company to handle that rollout as we’re not enjoying playing with AWS ourselves. Hope to be able to report (or speak!) soon!

    Comment by John (other John) — December 6, 2012 @ 11:39 pm

  13. look forward to it! The latency in the amazon cloud does not bode well for anything related to voice.. ANYTHING latency sensitive should stay out of clouds in general.

    Comment by Nate — December 18, 2012 @ 11:12 am

  14. Where?? I went poking around their site and could not find anything. I know if your an employee of a licensed company you can get it but otherwise… can you point me to a download link?


    Comment by Nate — December 18, 2012 @ 11:13 am

  15. dang, Nate, sorry the holidays distracted me, I’ll go find those ISO downloads! I found them before the holiday, so pretty sure I wasn’t dreaming it all up!

    whoop de doo, think this was one of them:

    and there’s another out there, different SKU, will find when I return.

    hope you had a top one, Nate!

    all best ~ j

    Comment by John (other John) — January 1, 2013 @ 10:00 pm

  16. totally agree with the latency point. I am plain unsure that a CDN can really do much when it has to point back, without serious bespoke engineering. I appreciate what CloudFlare are doing, for modest money, but unless I A/B test against just sticking a server next to a really well peered and transited router, I am not so sure. Basically, I think the pitch is “BGP is scary, so pay us to let you forget it.” think that Internap (IIRC) were doing something similar, just with expensive iron. Bit too spoiled, even at home, now, for connectivity, so actually gogt to the point where rendering was slowing things down! Happy New Year!

    Comment by John (other John) — January 1, 2013 @ 10:09 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress