TechOpsGuys.com Diggin' technology every day

November 12, 2010

Red Hat jacks up RHEL pricing

Filed under: linux,News — Nate @ 10:11 am

I didn’t think they would do this, but Red Hat, along with RHEL 6 introduced some pretty dramatic price hikes.

They seem to have done away with the “unlimited socket” licensing and have at least two tiers – two socket and four socket.

What used to cost $2,499 for Red Hat Advanced Server Premium for unlimited sockets, now costs $6,498 for four sockets, a 260% increase.

That is very Oracle-esque, maybe even worse than Oracle, the biggest hikes I recall Oracle doing was in the 30-50% range. Wonder if there will be any push back from customers.

They don’t seem to mention socket licensing beyond 4 sockets.

November 11, 2010

RHEL 6 Launched

Filed under: linux,News — Nate @ 9:04 pm

I didn’t even notice it, as The Register put it, it was a very quiet launch. While I have been using Debian on my home systems for more than twelve years now, I do much prefer to use Red Hat Enterprise at work.

And RHEL 6 looks like a pretty decent upgrade

  • Significantly improved power management (aka lower cpu usage for idle VMs) – hello higher consolidation ratios
  • Hot add CPU and memory (wish there was hot remove – if there is I don’t see it mentioned)
  • 85% increase in number of packages in the distribution – yay, maybe there will be a lot less things I will have to compile on my own

Sorry I still can’t help but laugh at the scalability claims

Red Hat Enterprise Linux 6 has been designed to deliver performance and scalability without sacrificing data integrity. It allows scaling to 4,096 CPUs and 64 terabytes of RAM, providing a solid foundation for supporting upcoming generations of hardware.

It is interesting that the max file system size for the ext4 file system is the same as ext3 – 16TB. Seems kind of dinky.

XFS goes to 100TB which also seems small, maybe just “tested” limits, I would expect XFS to scale far higher than that given it’s SGI heritage. The XFS documentation says for 64-bit Linux you can go to 18 Exabytes, which I think is just as crazy as Red Hat’s CPU claims but as long as you can safely do a few hundred TB that is more than enough for these days I think.

I can’t imagine anyone committing a petabyte or more to a single file system for a good long while at least.

I’ll let others play with KVM until at least RHEL 7, until then it’s VMware for me.

10% Tale of two search engines

Filed under: News,Random Thought — Tags: , — Nate @ 8:41 pm

Saw! an! article! today! and! thought! of! a! somewhat! sad! situation,! at! least! for! those! at! Yahoo!

Not long ago, Google announced they would be giving every employee in the company a 10% raise starting January 2011. One super bad ass engineer is apparently going to get a $3.5M retention bonus to not go to the competition. Lucky for him perhaps that Google is based in California and non competes are not enforceable in California.

Now Yahoo! has announced somewhat of the opposite, no raises, in fact they are going to give the axe to 10% of their employees.

It’s too bad that Yahoo! lost it’s way so long ago. There was a really good blog post about what went wrong with Yahoo! Going back  more than a decade, really interesting insight into the company.

Extreme VMware

Filed under: Networking,Virtualization — Tags: , — Nate @ 7:29 pm

So I was browsing some of the headlines of the companies I follow during lunch and came across this article (seems available on many outlets), which I thought was cool.

I’ve known VMware has been a very big happy user of Extreme Networks gear for a good long time now though I wasn’t aware of anything that was public about it, at least until today. It really makes me feel good that despite VMware’s partnerships with EMC and NetApp that include Cisco networking gear, at the end of the day they chose not to run Cisco for their own business.

But going beyond even that it makes me feel good that politics didn’t win out here, obviously the people running the network have a preference, and they were either able to fight, or didn’t have to fight to get what they wanted. Given VMware is a big company and given their big relationship with Cisco I would kind of think that Cisco would try to muscle their way in. Many times they can succeed depending on the management at the client company, but fortunately for the likes of VMware they did not.

SYDNEY, November 12. Extreme Networks, Inc., (Nasdaq: EXTR) today announced that VMware, the global leader in virtualisation and cloud infrastructure, has deployed its innovative enterprise, data centre and Metro Ethernet networking solutions.

VMware’s network features over 50,000 Ethernet ports that deliver connectivity to its engineering lab and supports the IT infrastructure team for its converged voice implementation.

Extreme Networks met VMware’s demanding requirements for highly resilient and scalable network connectivity. Today, VMware’s thousands of employees across multiple campuses are served by Extreme Networks’ leading Ethernet switching solutions featuring 10 Gigabit Ethernet, Gigabit Ethernet and Fast Ethernet, all powered by the ExtremeXOS® modular operating system.

[..]

“We required a robust, feature rich and energy efficient network to handle our data, virtualised applications and converged voice, and we achieved this through a trusted vendor like Extreme Networks, as they help it to achieve maximum availability so that we can drive continuous development,” said Drew Kramer, senior director of technical operations and R&D for VMware. “Working with Extreme Networks, from its high performance products to its knowledgeable and dedicated staff, has resulted in a world class infrastructure.”

Nice to see technology win out for once instead of back room deals which often end up screwing the customer over in the long run.

Since I’m here I guess I should mention the release of the X460 series of switches which came out a week or two ago, intended to replace the now 4-year old X450 series(both “A” and “E”). Notable differences & improvements include:

  • Dual hot swap internal power supplies
  • User swappable fan tray
  • Long distance stacking over 10GbE – up to 40 kilometers
  • Clear-Flow now available when the switches are stacked (prior hardware switches could not be stacked to use Clear-Flow
  • Stacking module is now optional (X450 it was built in)
  • Standard license is Edge license (X450A was Advanced Edge) – still software upgradable all the way to Core license (BGP etc). My favorite protocol ESRP requires Advanced Edge and not Core licensing.
  • Hardware support for IPFIX, which they say is complimentary to sFlow
  • Lifetime hardware warranty with advanced hardware replacement (X450E had lifetime, X450A did not)
  • Layer 3 Virtual Switching (yay!) – I first used this functionality on the Black Diamond 10808 back in 2005, it’s really neat.

The X460 seems to be aimed at the mid to upper range of GbE switches, with the X480 being the high end offering.

November 9, 2010

New NetApp boxes

Filed under: Storage — Nate @ 8:19 pm

So it looks like NetApp launched some beefy new systems yesterday, though I got to say if I was a customer of theirs I would feel kind of cheated on the 3200 series systems since they have stuck to dual core processors, when quad core has been available forever. In the “world of Intel” in my eyes there’s no excuse to release anything that’s not at least quad core unless your trying to squeeze your customers for every last bit (which I’m sure they are…).

Companies like NetApp could take a hint from someone like Citrix, who has a few Netscaler load balancers that they software rate limit the throughput but give you the same hardware as the higher end boxes. So take the 17500 model rated for 20Gbps, you can software upgrade that to more than double the throughput to 50Gbps. But the point isn’t the increased throughput via the software upgrade. The point is having the extra CPU horsepower on the smaller end box so that you can enable more CPU intensive features without incurring a noticeable performance hit because you have so much headroom on the system CPU wise.

NetApp introduced compression as one of their new features(I think it’s new, maybe wrong). That is of course likely to be a fairly CPU intensive operation. If they had quad or hex core CPUs in there, you could do a lot more, even if they limited your IOPS or throughput to X amount. Maybe they don’t have a good way of artificially rate limiting.

But even without rate limiting, it costs them a trivial amount of money to put quad core processors, they just want to squeeze their customers.

Even 3PAR put quad core processors in their F400 system more than a year ago. This is despite the Intel CPUs not doing much work on the 3PAR side, most of the work is done by their Gen3 ASIC. But they realize it’s a trivial cost to put in the beefier processor so they do it.

Their new 6200 series controllers do have quad core processors, among other improvements I’m sure. The previous 6000 series was quad socket. (in case your wondering where I’m getting these processors stats from it’s from the SPEC disclosure)

NetApp was fast to post both SPEC SFS results for their 3200 and 6200 series, as well as SPC-1E results for their 3200.

All in all very impressive results for SPEC SFS, very efficient results for SPC-1, both heavily assisted by 1TB of their flash cache. Interestingly enough at least on the SPC-1 side since full cost disclosures are there, the cost per usable TB and cost per IOP still doesn’t match that of the F400 (which has many more drives, and running RAID 1+0, and more than a year old so would consider the F400 at a great disadvantage but still wins out). SPC-1E isn’t a full SPC-1 test though, it’s more about power efficiency than raw performance. So time will tell if they do a “regular” SPC-1 test, their SPC-1E IOPS is about the same as their 3170, and the 3270 has much faster CPUs so I’d think it’s pretty safe to say that the controllers have capacity to go beyond 68,000 IOPS.

Nice upgrade for their customers in any case.

 

Next Gen Opterons — to 20 cores and beyond?

Filed under: News — Tags: , , — Nate @ 7:37 pm

So I came across this a short time ago, but The Register has a lot more useful information here.

From AMD

The server products (“Interlagos” and “Valencia”) will first begin production in Q2 2011, and we expect to launch them in Q3 2011. [This includes the Opteron 6100 socket compatible 16-core Opteron 6200]

[..]

Since Bulldozer is designed to fit into the same power/thermal envelope as our current AMD Opteron™ 6100/4100 series processors we obviously have some new power tricks up our sleeve.  One of these is the new CC6 power state, which powers down an entire module when it is idle. That is just one of the new power innovations that you’ll see with Bulldozer-based processors.

[..]

We have disclosed that we would include AMD Turbo CORE technology in the past, so this should not be a surprise to anyone. But what is news is the uplift – up to 500MHz with all cores fully utilized. Today’s implementations of boost technology can push up the clock speed of a couple of cores when the others are idle, but with our new version of Turbo CORE you’ll see full core boost, meaning an extra 500MHz across all 16 threads for most workloads.

[..]

We are anticipating about a 50% increase in memory throughput with our new “Bulldozer” integrated memory controller.

From The register

Newell showed off the top-end “Terramar” Opteron, which will have up to 20 of a next-generation Bulldozer cores in a single processor socket, representing a 25 percent boost in cores from the top-end Interlagos parts, and maybe a 35 to 40 per cent boost in performance if the performance curve stays the same as the jump from twelve-core “Magny-Cours” Opteron 6100s to the Interlagos chips.

[..]

That said, AMD is spoiling for a fight about chip design in a way that it hasn’t been since the mid-2000s.

[..]

with Intel working on its future “Sandy Bridge” and “Ivy Bridge” Xeon processors for servers, and facing an architecture shift in the two-socket space in 2011 that AMD just suffered through in 2010.

Didn’t Intel just go through an architecture shift in the two socket space last year with the Xeon 5500s and their integrated memory controller? And they are shifting architectures again so soon? Granted I haven’t really looked into what these new Intel things have to offer.

I suppose my only question is, will VMware come up with yet another licensing level to go beyond 12 cores per socket? It’s kind of suspicious that both vSphere Advanced and Enterprise plus are called out at a limit of 12 cores per socket.

November 6, 2010

The cool kids are using it

Filed under: Random Thought — Tags: , , — Nate @ 8:24 pm

I just came across this video, which is animated, involves a PHP web developer ranting to a psychologist about how stupid the entire Ruby movement is. It’s really funny.

I remember being in a similar situation a few years ago, the company had a Java application which drove almost all of the revenue of the company(90%+), and a perl application that they acquired from a ~2 person company and were busy trying to re-write it in Java.

Enter stage left: Ruby. At that point (sometime in 2006/2007), I honestly don’t think I had ever heard of Ruby before. But a bunch of the developers really seemed to like it, specifically the whole Ruby on Rails thing. We ran it on top of Apache with fastcgi. It really didn’t scale well at all (for fairly obvious reasons that are documented everywhere online). As time went on the company lost more and more interest in the Java applications and wanted to do everything in Ruby. It was cool (for them). Fortunately scalability was never an issue for this company since they had no traffic. At their peak they had four web servers, that on average peaked out at about 30-35% CPU.

It was a headache for me because of all of the modules they wanted to install on the system, and I was not about to use “gem install” to install those modules(that is the “ruby way” I won’t install directly from CPAN either BTW), I wanted proper version controlled RPMs. So I built them, for the five different operating platforms we supported at the time (CentOS 4 32/64bit, CentOS 5 32/64bit Fedora Core 4 32-bit — we were in transition to CentOS 5 32/64-bit). Looking back at my cfengine configuration file there was a total of 108 packages I built while I was there to support them, and it wasn’t a quick task to do that.

Then add to the fact that they were running on top of Oracle (which is a fine database IMO), mainly because that was what they had already running with their Java app. But using Oracle wasn’t the issue — the issue was their Oracle database driver didn’t support bind variables. If you have spent time with Oracle you know this is a bad thing. We used a hack which involved setting a per-session environment variable in the database to force bind variables to be enabled, this was OK most of the time, but it did cause major issues for a few months when a bad query got into the system, caused the execution plans to get out of whack and massive latch contention. The fastest way to recover the system was to restart Oracle. The developers, and my boss at the time were convinced it was a bug in Oracle. I was convinced it was not because I had seen latch contention in action several times in the past. After a lot of debugging the app and the database in consultation with our DBA consultants they figured out what the problem was — bad queries being issued from the app. Oracle was doing exactly what they told it to do, even if it means causing a big outage. Latch contention is one of the performance limits of Oracle that you cannot solve by adding more hardware. It seems like it could be at first because the results of it are throughput drops to the floor, and CPUs go to 100% usage instantly.

At one point to try to improve performance and get rid of memory leaks I migrated the Ruby apps from fastcgi to mod_fcgid. Which had a built in ability to automatically restart it’s threads after they had served X number of requests. This worked out great, really helped improve operations. I don’t recall if it had any real impact on performance but because the memory leaks were no longer a concern that was one less thing to worry about.

Then one day we got in some shiny new HP DL380 G5s with dual proc quad core processors with either 8 or 16GB of memory. Very powerful, very nice servers for the time. So what was the first thing I tried? I wanted to try out 64-bit, be able to take better advantage of the larger amount of memory. So I compiled our Ruby modules for 64-bit, installed a 64-bit CentOS 5.2 I think it was at the time(other production web servers were running CentOS 5.2 32-bit), installed 64-bit Ruby etc. Launched the apps, from a functional perspective they worked fine. But from a practical perspective it was worthless. I enabled the web server in production and it immediately started gagging on it’s own blood, load shot through the roof, requests were slow as hell. So I disabled it, and things returned to normal. Tried that a few more times and ended up giving up — went back to 32-bit. The 32-bit system could handle 10x the traffic of the 64-bit system. Never found out what the issue was before I left the company.

From an operational perspective, my own personal preference for web apps is to run Java. I’m used to running Tomcat myself but really the container matters less to me. I like war files, it makes deployment so simple. And in the Weblogic world I liked ear files (I suspect it’s not weblogic specific it’s just the only place I’ve ever used ear files). One archive file that has everything you need built into it. Any extra modules etc are all there. I don’t have to go compile anything, install a JVM, install a container and drop a single file to run the application. OK maybe some applications have a few config files (one I used to manage had literally several hundred XML config files — poor design of course).

Maybe it’s not cool anymore to run Java I don’t know. But seeing this video reminded me of those days when I did have to support Ruby on production and pre-production systems, it wasn’t fun, or cool.

November 4, 2010

Chicken and the egg

Filed under: Random Thought,Storage,Virtualization — Tags: , , , , , , — Nate @ 5:24 pm

Random thought time! –  came across an interesting headline on Chuck’s Blog – Attack of the Vblock Clones.

Now I’m the first to admit I didn’t read the whole thing but the basic gist he is saying if you want a fully tested integrated stack (of course you know I don’t like these stacks they restrict you too much, the point of open systems is you can connect many different types of systems together and have them work but anyways), then you should go with their VBlock because it’s there now, and tested, deployed etc. Others recently announced initiatives are responses to the VBlock and VCE, Arcadia(sp?) etc.

I’ve brought up 3cV before, something that 3PAR coined back almost 3 years ago now. Which is, in their words “Validated Blueprint of 3PAR, HP, and VMware Products Can Halve Costs and Floor Space”.

And for those that don’t know what 3cV is, a brief recap –

The Elements of 3cV
3cV combines the following products from 3PAR, HP, and VMware to deliver the virtual data center:

  • 3PAR InServ Storage Server featuring Virtual Domains and thin technologies—The leading utility storage platform, the 3PAR InServ is a highly virtualized tiered-storage array built for utility computing. Organizations creating virtualized IT infrastructures for workload consolidation use the 3PAR InServ to reduce the cost of allocated storage capacity, storage administration, and the SAN infrastructure.
  • HP BladeSystem c-Class—The No. 1 blade infrastructure on the market for datacenters of all sizes, the HP BladeSystem c-Class minimizes energy and space requirements and increases administrative productivity through advantages in I/O virtualization, power and cooling, and manageability. (1)
  • VMware Infrastructure—Infrastructure virtualization suite for industry-standard servers. VMware Infrastructure delivers the production-proven efficiency, availability, and dynamic management needed to build the responsive data center.

Sounds to me that 3cV beat VBlock to the punch by quite a ways. It would have been interesting to see how Dell would of handled the 3cV solution had they managed to win the bidding war, given they don’t have anything that competes effectively with c-Class. But fortunately HP won out so 3cV can be just that much more official.

It’s not sold as a pre-packaged product I guess you could say, but I mean how hard is it to say I need this much CPU, this much ram, this much storage HP go get it for me. Really it’s not hard. The hard part is all the testing and certification. Even if 3cV never existed you can bet your ass that it would work regardless. It’s not that complicated, really. Even if Dell managed to buy 3PAR and kill off the 3cV program because they wouldn’t want to directly promote HP’s products, you could still buy the 3PAR from Dell and the blades from HP and have it work. But of course you know that.

The only thing missing from 3cV is I’d like a more powerful networking stack, or at least sFlow support. I’ll take Flex10 (or Flexfabric) over Cisco any day of the week but I’d still like more.

I don’t know why this thought didn’t pop into my head until I read that headline, but it gave me something to write about.

But whatever, that’s my random thought of the day/week.

October 28, 2010

Compellent beats expectations

Filed under: News,Random Thought,Storage — Tags: , — Nate @ 11:10 am

Earlier in the year Compellent‘s stock price took a big hit following lower expectations for sales and stuff, a bunch of legal stuff followed that, it seems yesterday they redeemed themselves though with their stock going up nearly 33% after they tripled their profits or something.

I’ve had my eye on Compellent for a couple of years now, don’t remember where I first heard about them. They have similar technology to 3PAR, just it’s implemented entirely in software using Intel CPUs as far as I know vs 3PAR leveraging ASICs (3PAR has Intel CPUs too but they aren’t used for too much).

I have heard field reports that because of this that their performance is much more on the lower end of things, they have never published a SPC-1 result and I don’t know anyone that uses them so don’t know how they really perform.

They seem to use the same Xyratex enclosures that most everyone else uses. Compellent’s controllers do seem to be somewhat on the low end of things, I really have nothing other to go on other than cache. With their high end controller coming in at only 3.5GB of cache (I assume 7GB mirrored for a pair of controllers?) it is very light on cache. The high end has a dual core 3.0Ghz CPU.

The lower amount of cache combined with their software-only design and only two CPUs per controller and the complex automated data movement make me think the systems are built for the lower end and not as scalable, but I’m sure perfectly useful for the market they are in.

Would be nice to see how/if their software can scale if they were to put say a pair of 8 or 12 core CPUs in their controllers. After all since they are leveraging x86 technology performing such an upgrade should be pretty painless! Their controller specs have remained the same for a while now(as far back as I can remember). The bigger CPUs will use more power, but from a storage perspective I’m happy to give a few hundred more watts if I can get 5x+ the performance, don’t have to think once, yet alone twice.

They were, I believe the first to have automagic storage tiering and for that they deserve big props, though again no performance numbers posted (that I am aware of) that can illustrate the benefits this technology can bring to the table. I mean if anybody can prove this strategy works it should be them right? On paper it certainly sounds really nice but in practice I don’t know, haven’t seen indications that it’s as ready as the marketing makes it out to be.

My biggest issue with automagic storage tiering is how fast the array can respond to “hot” blocks and optimize itself, which is why I think from a conceptual perspective I really like the EMC Fast Cache approach more (they do have FAST LUN and sub LUN tiering too). Not that I have any interest in using EMC stuff but they do have cool bits here and there.

Maybe Compellent the next to get bought out (as a block storage company yeah I know they have their zNAS), I believe from a technology standpoint they are in a stronger position than the likes of Pillar or Xiotech.

Anyway that’s my random thought of the day

October 26, 2010

Rust in peace

Filed under: News — Nate @ 8:53 am

So one of of the former TechOpsGuys Tycen emailed our little team of people yesterday mentioning that the company where we all met (Dave, Jason, Nate & Tycen) is now dead and buried according to an article on TechFlash. We were all on the same operations team in case it wasn’t obvious.

Recruiting.com, the Seattle online recruiting startup formerly known as Jobster, has quietly been sold in a deal of unknown size to Phoenix, Arizona-based Jobing, TechFlash has learned. The acquisition, which closed in July, marks the final chapter for one of Seattle’s most heavily funded Internet companies of the past decade. Founded in 2004 by Internet entrepreneur Jason Goldberg, Recruiting.com raised $55 million from a slate of investors that included Ignition Partners, Reed Elsevier Ventures, Trinity Ventures, Mayfield Fund and others.

Where did the money go? I don’t know, we were really efficient in IT, and I think development was pretty good, so I can only think that it was the less technical areas of the company that spent most of the cash. $55 million doesn’t seem like enough to say “one of the most heavily funded”. Doesn’t seem like much at all. I know it was a different era but I remember at Freeinternet.com (headquartered in Federal Way, near Seattle) they were spending $7 million/month on bandwidth they weren’t even using (they didn’t know they had it). That is just one of the little factoids.

The strategies the company had while I was there were typical? Nothing short of their original strategy made any sense to me and unfortunately for them they kept pulling resources out of that original strategy (the one generating the majority of the revenue but it wasn’t shiny). We knew we were in trouble when we were denied the right to remove email addresses from our databases that were bouncing back causing us to get blacklisted on mail servers due to excessive bouncing. It would hurt our user count. Well those users don’t exist anymore!!

Jobster certainly provided me with several opportunities to round out some of my skills and experience. I had learned a lot prior to that but often worked with dedicated people whether it was networking people, storage people etc. So while I knew what I was doing it was nice to be able to build things up from scratch. Did a data center move while I was there, moving out of a facility that was plagued with power outages in cabinets spread out all over the place to a better facility in a consolidated cage. Got a lot more hands on with our Oracle Databases as well, at previous companies we had dedicated Oracle folks and they pretty much ran everything. Speaking of Oracle even transitioned the company from Enterprise Edition to Standard Edition which of course saved a ton of greenbacks.

I went looking for my stock certificates I bought a few shares after I left, the price was cheap (~$0.15) just in case. For some reason I could not find them, maybe they didn’t send them to me I don’t remember it was a while ago. Not that I think the stock has any value but wanted to post a picture of what they looked like for some color 🙂

It was an interesting place to work for at the time, I joined shortly after they launched their main product and it was pretty fast downhill from there. What can we say, we built the system to scale to the growth expectations that they had and that growth never materialized.

At the peak we had almost three racks of production equipment (15kW active 15kW backup power), in my mac daddy 47U 28″ wide 49″ deep Rittal TS-8 racks. Those racks are bad ass I gotta say. I could only fit three of them in a five rack cage (but due to power we had more than enough space in three). The cable management ROCKED (each server had at least 5 cables, which adds up fast).  The extra 4″ of width made all the difference. The Servertech PDUs were great too, loved the integrated environmental sensors, rear door fans to exhaust the hot air away from the equipment. I love big racks! (couldn’t resist)

The different facility was really nice as well the only downside is it doesn’t support high density stuff.  I was hosted at the same place at the company before that and due to power constraints we had to have 8 feet of space in between our rows of racks. Not that I minded too much 8 feet was a lot of area to stretch out, put tables and shelves in for storage etc. With ~30 foot ceilings it’s one of the few data centers I’ve been to that didn’t make me feel claustrophobic.

My strategy for the company at the time was to make their main product as good as possible and then partner with someone like linkedin to leverage their network with Jobster’s technology. Jobster’s main product was never visible to the public it was a subscription only product, so most probably never saw the technology that the company was built on (initially at least). Their strategy was OOH, SHINY! rinse and repeat (at least 3 times while I was there).

« Newer Posts — Older Posts »

Powered by WordPress