TechOpsGuys.com Availability matters.

HP BL685c G7 Launched – Opteron 6100

TechOps Guy: Nate

I guess my VMware dream machine will remain a dream for now, HP launched their next generation G7 Opteron 6100 blades today, and while still very compelling systems, after the 6100 launched I saw the die size had increased somewhat (not surprising), it was enough to remove the ability to have 4 CPU sockets AND 48 memory slots on one full height blade.

• Specifications for HP BL685c G6 (previous generation)
• Specifications for HP BL495c G6 (previous generation)
• Specifications for HP BL685c G7
• Specifications for HP BL485c G7

Still a very good comparison illustrating the elimination of the 4P tax, that is eliminating the premium associated with quad socket servers. If you configure a BL485c G7 with 2×12-core CPUs and 128GB of memory(about $16,000), vs a BL685c G7 with 256GB of memory and the 4×12-core CPUs (about $32,000), the cost is about the same, no premium.

By contrast configuring a BL685c G6 with six core CPUs (e.g. half the number of cores as the G7), same memory, same networking, same fiber channel, the cost is roughly $52,000.

These have new Flex Fabric 2 NICs, which from the specs page seem to indicate they include iSCSI or FCoE support (I assume some sort of software licensing needed to unlock the added functionality? though can’t find evidence of it). Here is a white paper on the Flex Fabric stuff, from what I gather it’s just an evolutionary step of Virtual Connect. Myself of course have never had any real interest in FCoE (search the archives for details), but nice I suppose that HP is giving the option to those that do want to jump on that wagon.

Sea Micro launches 512 core Atom server

TechOps Guy: Nate

An article from our friends at The Register talks about a new server design to hit the market. A very innovative solution from a recently decloaked stealth startup Sea Micro based on the Intel Atom processor, called the SM 10000.

Looks to be targetted at the hyperscale arena, SGI tried something similar to this last year with their Microslice design, though it’s not nearly as efficient as this box is.

The SM1000 is a fairly radical departure from current designs, perhaps the closest design I’ve come across to this SM monster is a design from Dell’s DCS division a few years ago that The Register reported on. This goes several steps beyond that by including in a single 10U chassis:

• Up to 512 Atom CPUs each with up to 2GB memory
• Up to 64  x 2.5″ disks
• Integrated ethernet switching and load balancing
• Integrated terminal server
• Virtualized I/O

This is targetted to a specific application – mainly web serving. The massive amount of parallelism in the system combined with the low power foot print (a mere 4W/server) can provide a high amount of throughput for many types of web applications. The ability to have SSDs in the system allow high I/O rates for smaller data sets.

From one of their white papers:

[..]hardware-based CPU I/O virtualization enables SeaMicro to eliminate 90 percent of the components from the server and to shrink the motherboard to the size of a credit card. Hundreds of these low-power, card-sized computational units are tied together with a supercomputer-style fabric to create a massive array of independent but linked computational units. Work is then distributed over these hundreds of CPUs via hardware- and software-based load-balancing technology that dynamically directs load to ensure that each of the CPUs is either in its most efficient zone of performance or is sleeping. The key technologies reside in three chips of SeaMicro’s design, one ASIC and two FPGAs, and in the management, routing, and load-balancing software that directs traffic across the fabric.

It’s clearly targeted at the scale out web serving market, the likes of Google, Facebook, Yahoo. These aren’t general purpose servers, I saw some stupid posts on Slashdot mentioning trying to run VMware on or something on top of this. The system is virtualized at the hardware level, there’s no need for a software hypervisor running on top.

From another white paper, talking about the virtualized disk technology:

The SeaMicro SM10000 can be configured with 0 to 64 2.5 inch SATA hard disk drives (HDD) or solid state drives (SSD). The 512 CPUs in the system can be allocated portions of a disk or whole disks. A physical disk (HDD or SSD) can be divided into multiple virtual disks – from 2GB to the maximum capacity of the disk – and assigned to one or more CPUs. Data resiliency is maintained by marking a disk to be part of a RAID pool or by assigning multiple disks to a CPU. The system can be configured to run with or without disk, ensuring the flexibility to appropriately provision storage for the desired applications

My only questions at this stage would be:

• How well does it work? Not knowing the internals of where they got their ethernet switching or load balancing technology from, or even RAID technology.
• Their CPU of choice is 32-bit. For many workloads this is fine, though many others need 64bit.
• Questions on how the shared disks work – you have the ability to take a SSD for example and put shared application code on a read only portion of the disk that can be read by as many servers in the system as you want, I suppose to take maximum advantage of the technology in the system you may have to make some changes to your application(s), it would be cool if they offered the ability to have the shared disk be writable by more than one system, using a cluster file system or something. Maybe this is possible I don’t know.

A Base configuration starts at about $139,000 according to The Register. No mention of what that includes though.

Certainly seems to be a system that has a lot of promise for the market it is targetted towards!

Did I ever mention I love technology?

Found a use for the cloud

TechOps Guy: Nate

Another interesting article on Datacenter Knowledge and mentioned the U.S. Government’s use of the Terremark cloud, I recall reading about it briefly when it first launched but seeing the numbers again made me do another double take.

”One of the most troubling aspects about the data centers is that in a lot of these cases, we’re finding that server utilization is actually around seven percent,” Federal Chief Information Officer Vivek Kundra said

[..]

Yes, you read that correctly. A government agency was going to spend $600,000 to set up a blog.

[..]

The GSA previously paid $2.35 million in annual costs for USA.gov, including $2 million for hardware refreshes and software re-licensing and $350,000 in personnel costs, compared to the $650,000 annual cost to host the site with Terremark.

For $650k/yr I bet the site runs on only a few servers(dozen or less) and has less than a TB of total disk space.

New IBM blades based on Intel 7500 announced

TechOps Guy: Nate

The Register had the scoop a while back, but apparently today they were officially announced. IBM did some trickery with the new 7500 series Intel Xeons to accomplish two things:

• Expand the amount of memory available to the system
• Be able to “connect” two dual socket blades to form a single quad socket system

Pretty creative, though the end result wasn’t quite as impressive as it sounded up front. Their standard blade chassis is 9U and has 14 slots on it.

• Each blade is dual socket, maximum 16 cores, and 16 DIMMs
• Each memory extender offers 24 additional DIMMs

So for the chassis as a whole your talking about 7 dual socket systems with 40 DIMMs each. Or 3 quad socket systems with 80 DIMMs each, and 1 dual socket with 40.

Compared to an Opteron 6100 system, which you can get 8 quad socket systems with 48 DIMMs each in a single enclosure(granted such a system has not been announced yet but I am confident it will be).

• Intel 7500-based system: 112 CPU cures (1.8Ghz), 280 DIMM slots – 9U
• Opteron 6100-based system: 384 CPU cores (2.2Ghz), 384 DIMM slots – 10U

And the price of the IBM system is even less impressive -

In a base configuration with a single four-core 1.86 GHz E7520 processor and 8 GB of memory, the BladeCenter HX5 blade costs $4,629. With two of the six-core 2 GHz E7540 processors and 64 GB of memory, the HX5 costs $15,095.

They don’t seem to show pricing for the 8 core 7500-based blade, and say there is no pricing or ETA on the arrival of the memory extenders.

They do say this which is interesting (not surprising) -

The HX5 blade cannot support the top-end eight-core Xeon 7500 parts, which have a 130 watt thermal design point, but it has been certified to support the eight-core L7555, which runs at 1.86 GHz, has 24 MB of L3 cache, and is rated at 95 watts.

I only hope AMD has enough manufacturing capacity to keep up with demand, Opteron 6100s will wipe the floor with the Intel chips on price/performance (for the first time in a while).

Opteron 6100s are here

TechOps Guy: Nate

UPDATED I’ve been waiting for this for quite some time, finally the 12-core AMD Opteron 6100s have arrived. AMD did the right thing this time by not waiting to develop a “true” 12-core chip and instead bolted a pair of CPUs together into a single package. You may recall AMD lambasted Intel when it released it’s first four core CPUs a few years ago(composed of a pair of two-core chips bolted together), a strategy that paid off well for them, AMD’s market share was hurt badly as a result, a painful lesson which they learned from.

For me I’d of course rather have a “true” 12-core processor, but I’m very happy to make do with these Opteron 6100s in the meantime, I don’t want to have to wait another 2-3 years to get 12 cores in a socket.

Some highlights of the processor:

• Clock speeds ranging from 1.7Ghz(65W) to 2.2Ghz(80W), with a turbo boost 2.3Ghz model coming in at 105W
• Prices ranging from $744 to $1,396 in 1,000-unit quantities
• Twelve-core and Eight–core, L2 – 512K/core, L3 – 12MB of shared L3 Cache
• Quad-Channel LV & U/RDDR3, ECC, support for on-line spare memory
• Supports up to 3 DIMMs/channel, up to 12 DIMMS per CPU
• Quad 16-bit HyperTransport™ 3 technology (HT3) links, up to 6.4 GT/s per link (more than triple HT1 performance)
• AMD SR56×0 chipset with I/O Virtualization and PCIe® 2.0
• Socket compatibility with planned AMD Opteron™ 6200 Series processor.(16 cores?)
• New advanced idle states allowing the processor to idle with less power usage than the previous six core systems (AMD seems to have long had the lead in idle power conservation).

The new I/O virtualization looks quite nice as well – AMD-V 2.0, from their site:

Hardware features that enhance virtualization:

• Unmatched Memory Bandwidth and Scalability – Direct Connect Architecture 2.0 supports a larger number of cores and memory channels so you can configure robust virtual machines, allowing your virtual servers to run as close as possible to physical servers.
• Greater I/O virtualization efficiencies –I/O virtualization to help increase I/O efficiency by supporting direct device assignment, while improving address translation to help improve the levels of hypervisor intervention.
• Improved virtual machine integrity and security –With better isolation of virtual machines through I/O virtualization, helps increase the integrity and security of each VM instance.
• Efficient Power Management – AMD-P technology is a suite of power management features that are designed to drive lower power consumption without compromising performance. For more information on AMD-P, click here
• Hardware-assisted Virtualization – AMD-V technology to enhance and accelerate software-based virtualization so you can run more virtual machines, support more users and transactions per virtual machine with less overhead. This includes Rapid Virtualization Indexing (RVI) to help accelerate the performance of many virtualized applications by enabling hardware-based VM memory management. AMD-V technology is supported by leading providers of hypervisor and virtualization software, including Citrix, Microsoft, Red Hat, and VMware.
• Extended Migration – a hardware feature that helps virtualization software enable live migration of virtual machines between all available AMD Opteron™ processor generations. For a closer look at Extended Migration, follow this link.

With AMD returning to the chipset design business I’m happy with that as well, I was never comfortable with Nvidia as a server chipset maker.

The Register has a pair of great articles on the launch as well, though the main one I was kind of annoyed I had to scroll so much to get past the Xeon news, which I don’t think they had to go out of their way to recap with such detail in an article about the Opterons, but oh well.

I thought this was an interesting note on the recent Intel announcement of integrated silicon for encryption -

While Intel was talking up the fact that it had embedded cryptographic instructions in the new Xeon 5600s to implement the Advanced Encryption Standard (AES) algorithm for encrypting and decrypting data, Opterons have had this feature since the quad-core “Barcelona” Opterons came out in late 2007, er, early 2008.

And as for performance -

Generally speaking, bin for bin, the twelve-core Magny-Cours chips provide about 88 per cent more integer performance and 119 per cent more floating point performance than the six-core “Istanbul” Opteron 2400 and 8400 chips they replace..

AMD seems geared towards reducing costs and prices as well with -

The Opteron 6100s will compete with the high-end of the Xeon 5600s in the 2P space and also take the fight on up to the 4P space. But, AMD’s chipsets and the chips themselves are really all the same. It is really a game of packaging some components in the stack up in different ways to target different markets.

Sounds like a great way to keep costs down by limiting the amount of development required to support the various configurations.

AMD themselves also blogged on the topic with some interesting tidbits of information -

You’re probably wondering why we wouldn’t put our highest speed processor up in this comparison. It’s because we realize that while performance is important, it is not the most important factor in server decisions.  In most cases, we believe price and power consumption play a far larger role.

[..]

Power consumption – Note that to get to the performance levels that our competitor has, they had to utilize a 130W processor that is not targeted at the mainstream server market, but is more likely to be used in workstations. Intel isn’t forthcoming on their power numbers so we don’t really have a good measurement of their maximum power, but their 130W TDP part is being beaten in performance by our 80W ACP part.  It feels like the power efficiency is clearly in our court.  The fact that we have doubled cores and stayed in the same power/thermal range compared to our previous generation is a testament to our power efficiency.

Price – This is an area that I don’t understand.  Coming out of one of the worst economic times in recent history, why Intel pushed up the top Xeon X series price from $1386 to $1663 is beyond me.  Customers are looking for more, not less for their IT dollar.  In the comparison above, while they still can’t match our performance, they really fall short in pricing.  At $1663 versus our $1165, their customers are paying 42% more money for the luxury of purchasing a slower processor. This makes no sense.  Shouldn’t we all be offering customers more for their money, not less?

In addition to our aggressive 2P pricing, we have also stripped away the “4P tax.” No longer do customers have to pay a premium to buy a processor capable of scaling up to 4 CPUs in a single platform.  As of today, the 4P tax is effectively $0. Well, of course, that depends on you making the right processor choice, as I am fairly sure that our competitor will still want to charge you a premium for that feature.  I recommend you don’t pay it.

As a matter of fact, a customer will probably find that a 4P server, with 32 total cores (4 x 8-core) based on our new pricing, will not only perform better than our competitor’s highest end 2P system, but it will also do it for a lower price. Suddenly, it is 4P for the masses!

While for the most part I am mainly interested in their 12-core chips, but I also see significant value in the 8 core chips, being able to replace a pair of 4 core chips with a single socket 8 core system is very appealing as well in certain situations. There is a decent premium on motherboards that need to support more than one socket. Being able to get 8, (and maybe even 12 cores) on a single socket system is just outstanding.

I also found this interesting -

Each one is capable of 105.6 Gigaflops (12 cores x 4 32-bit FPU instructions x 2.2GHz).  And that score is for the 2.2GHz model, which isn’t even the fastest one!

I still have a poster up on one of my walls back from 1995-1996 era on the world’s first Teraflop machine, which was -

The one-teraflops demonstration was achieved using 7,264 Pentium Pro processors in 57 cabinets.

With the same number of these new Opterons you could get 3/4ths of the way to a Petaflop.

SGI is raising the bar as well -

This means as many as 2,208 cores in a single rack of our Rackable™ rackmount servers. And in the SGI ICE Cube modular data center, our containerized data center environment, you can now scale within a single container to 41,760 cores! Of course, density is only part of the picture. There’s as much to be excited about when it comes to power efficiency and the memory performance of SGI servers using AMD Opteron 6100 Series processor technology

Other systems announced today include:

• HP DL165G7
• HP SL165z G7
• HP DL385 G7
• Cray XT6 supercomputer
• There is mention of a Dell R815 though it doesn’t seem to be officially announced yet. The R815 specs seem kind of underwhelming in the memory department, with it only supporting 32 DIMMs (the HP systems above support the full 12 DIMMs/socket). It is only 2U however. Sun has had 2U quad socket Opteron systems with 32 DIMMs for a couple years now in the form of the X4440, strange that Dell did not step up to max out the system with 48 DIMMs.

I can’t put into words how happy and proud I am of AMD for this new product launch, not only is it an amazing technological achievement, but the fact that they managed to pull it off on schedule is just amazing.

Congratulations AMD!!!

IBM partners with Red hat for KVM cloud

TechOps Guy: Nate

One question: Why?

IBM has bombarded the IT world for years now how they can consolidate hundreds to thousands of Linux VMs onto a single mainframe.

IBM has recently announced a partnership with Red hat to use KVM in a cloud offering. At first I thought, well maybe they are doing it to offer Microsoft applications as well, but that doesn’t appear to be the case:

Programmers who use the IBM Cloud for test and dev will be given RHEV to play with Red Hat Enterprise Linux or Novell SUSE Linux Enterprise Server images with a Java layer as they code their apps and run them through regression and other tests.

Let’s see, Linux and Java, why not use the mainframes to do this? Why KVM? As far as the end users are concerned it really shouldn’t matter, after all it’s java and linux.

Seems like a slap in the face to their mainframe division (I never bought into the mainframe/linux/VM marketing myself, I suppose they don’t either). I do remember briefly having access to a S390 running a SuSE VM about 10 years ago, it was..interesting.