TechOpsGuys.com Diggin' technology every day

September 21, 2012

Need New Colo! NOW

Filed under: Datacenter — Nate @ 1:36 pm

UPDATE – I have finished my move! woohoo. This is my first visit to this Hurricane Electric facility it’s not really what I expected but it is an, interesting place. It seems to be entirely shared hosting, well at least the room I am in right now. All of the racks are the same and things look fairly uniform. I can tell the racks are pretty old tech, reminds me of the racks I used to have at Internap. I wouldn’t want to host mission critical stuff here, or expensive stuff, but for my little 1U box it is OK. There is a cabinet one cabinet away from mine in my row where they removed the rear door to fit the equipment in. They have a C7000 blade enclosure  and tons of CAT5 cabling all over the place,  totally not secure, but I guess people here are mindful which is good. They look to be running what I assume is a 120V 20A circuit based on the size of the power cable coming out of the PDU, and there is a little meter there on the PDU itself that reads 20 …  The PDU is ziptied to the rear of the rack because there is no place to mount it in the rack (they had to extend the rack rails to the max to fit the blades in). Anyways off I go…

 

[UPDATED] My hosting provider broke the news to me yesterday (but didn’t see the email till today) that they are terminating their co-location on Oct 1st. So I have to get out and find somewhere else to put my server (which hosts this blog along with my email, my DNS etc..).

I reached out to Hurricane Electric where my current provider hosts, as well as another place called Silicon Valley Web Hosting, which seems decent. If you know of another provider in the bay area (I want it local in case I have hardware issues), please let me know!

This is what I have today:

  • 1U Server
  • 1 network feed
  • 5 static IPs
  • 100Mbit unlimited bandwidth
  • $100/mo

I don’t use much bandwidth, I haven’t kept track in a while but I suspect it’s well below 100GB/mo.

SV Web hosting looks to cost about $170/mo for at a facility they are hosted at down in San Jose. I think I’d be OK up to $190/mo, beyond that I’d have to think harder about making more compromises.

I’d like to be hosted in a good facility, and am totally willing to pay more for it – no fly by night operations, nobody that attracts a lot of spammers/DOS attack type stuff.

I don’t need 100% uptime, network outages here and there aren’t a big deal, hopefully power outages will be really rare. Oh and no facilities that run on flywheel UPSs now that I’m thinking about power. Redundant power would be nice (my server has it), not required, redundant network would be nice, not required.

So if you know of any good places I should check out let me know!

(for those of you who may not know I did try the cloud route for a year while I was in between servers – it really wasn’t for me – especially now that I have 4TB of usable storage on my system for backups and stuff)

thanks

UPDATE – after some hard thinking I decided to go with Hurricane Electric, the same building that I am hosted in now so the move should be pretty painless. For $200/mo I get 7U of space, with 240W of power and 100Mbit unmetered bandwidth. I think the extra space will give me some flexibility, at some point I plan to get a pair of these, and rotate them between home and colo, 1TB is more than enough, with my ~1Mbit upstream it would take me 4 months to upload 1TB(the best upstream I can get is “up to 5Mbps” for an extra $55/mo – currently my “up to 1.5Mbps” seems to tap out below 1.1Mbps), vs a 30 minute drive and probably 12 hours of data copying with this method. Then probably either move my backup Soekris OpenBSD firewall or get another one so I can better protect the ESXi and ipmi management interfaces on my server (the VM that runs this blog already sits behind a virtual OpenBSD firewall).

Longer term I can see building a new server to replace what I have, something that is bigger, supports more disks, but still efficient enough to fit in that 240W power envelope. Longer term still who knows maybe I will upgrade to 100Mbps cable modem for $160/mo (currently pay about $45 for 16Mbps), and just store everything in my own little cloud. Stream everything..

September 14, 2012

VMware suggests swapping as a best practice

Filed under: Virtualization — Tags: — Nate @ 11:26 am

Just came across this, and going through the PDF it says

Virtualization causes an increase in the amount of physical memory required due to the extra memory needed by ESXi for its own code and for data structures. This additional memory requirement can be separated into two components:
1. A system-wide memory space overhead for the VMkernel and various host agents (hostd, vpxa, etc.).

A new feature in ESXi 5.1 allows the use of a system swap file to reduce this memory overhead by up to 1GB when the host is under memory pressure.

That just scares me that the advocate setting up a swap file to reduce memory usage by up to 1GB. How much memory does the average VMware system have? Maybe 64GB today? So that could save 1.5% of physical memory, with the potential trade off of impacting storage performance (assuming no local storage) for all other systems in the environment.

Scares me just about as much as how 3PAR used to advocate their storage systems can get double the VM density per server because you can crank up the swapping and they can take the I/O hit (I don’t think they advocate this today though).

Now if you can somehow be sure that the system won’t be ACTIVELY swapping then it’s not a big deal, but of course you don’t want to actively swap really in any situation, unless your I/O is basically unlimited. You could go and equip your servers with say a pair of SSDs in RAID 1 to do this sort of swapping(remember it is 1GB). But it’s just not worth it. I don’t understand why VMware spent the time to come up with such a feature.

If anything the trend has been more memory in hosts not less, I’d imagine most serious deployments have well over 100GB of memory per host these days.

My best practice is don’t swap – ever. In the environments I have supported performance/latency is important so there is really no over subscription for memory, I’ve had one time where Vmware was swapping excessively at the host level, and to me it was a bug, but to Vmware it was a feature(there was tons of memory available on the host), I forgot the term but it was a documented behavior on how the hypervisor functions, just not commonly known I guess, and totally not obvious. The performance of the application obviously went in the toilet when this swapping was going on, it felt like the system was running on a 386 CPU.

Windows memory footprint is significantly different than that of Linux, Linux represents probably 98 or 99% of my VMs over the years.

Oh and that transparent page sharing VMware touts so much? I just picked one of my servers at random, 31 VMs and 147GB of memory in use, TPS is saving me a grand 3% of memory, yay TPS.

The cost of I/Os(to spinning rust, or even enterprise SSDs), unless your workload is very predictable and you do not have much active swapping, is just too much to justify the risk in allowing swap in any form in my experience. In fact the bulk of the VMs I run do have a local 500MB swap partition, enough for some very light swapping – but I’d rather have the VM fail & crash, then have it swap like crazy and take the rest of the systems down with it.

But that’s me

September 12, 2012

Data Center reminder: deploy environmental sensors

Filed under: Datacenter — Tags: — Nate @ 8:54 pm

I feel like I am almost alone in the world when it comes to deploying environmental sensors around my equipment. I first did it at home back around 2001 when I had a APC SmartUPS and put a fancy environmental monitoring card in it, which I then wrote some scripts for and tied it into MRTG.

A few years later I was part of a decently sized infrastructure build out that had a big budget so I got one of these, and 16 x environmental probes each with 200 foot cables (I think the probes+cables alone were about $5k(the longest cables they had at the time, which were much more expensive than the short ones, I wasn’t sure what lengths I needed so I just went all out), ended up truncating the ~3200 feet of cables down to around ~800 feet I suspect). I focused more on cage environmental than per rack, I would of needed a ton more probes if I had per rack. Some of the sensors went into racks, and there was enough slack on the end of the probes to temporarily position them anywhere within say 10 feet of their otherwise fixed position very easily.

The Sensatronics device was real nifty, so small, and yet it supported both serial and ethernet, had a real basic web server, was easily integrated to nagios (though at the time I never had the time to integrate it so relied entirely on the web server). We were able to prove to the data center at the time their inadequate cooling and they corrected it by deploying more vented tiles. They were able to validate the temperature using one of those little laser gun things.

At the next couple of companies I changed PDU manufacturers and went to ServerTech instead, many (perhaps all?) of their intelligent PDUs come with ports for up to two environmental sensors. Some of their PDUs require an add-on to get the sensor integration.

The probes are about $50 a piece and have about a 10 foot cable on them. Typically I’d have two PDUs in a rack and I’d deploy four probes (2 per PDU). Even though environmental SLAs only apply to the front of the racks, I like information so I always put two sensors in front and two sensors in rear.

I wrote some scripts to tie this sensor data into cacti (the integration is ugly so I don’t give it out), and later on I wrote some scripts to tie this sensor data into nagios (this part I did have time to do). So I could get alerts when the facility went out of SLA.

Until today the last time I was at a facility that was out of SLA was in 2009, when one of the sensors on the front of the rack was reporting 87 degrees. The company I was at during that point had some cheap crappy IDS systems deployed in each facility, and this particular facility had a high rate of failures for these IDSs. At first we didn’t think *too* much of it, then I had the chance to hook up the sensors and wow, was I surprised. I looked at the temperatures inside the switches and compared it to other facilities (can’t really extrapolate ambient temp from inside the switch), and confirmed it was much warmer there than at our other locations.

So I bitched to them and they said there was no problem, after going back and forth they did something to fix it – this was a remote facility – 5,000 miles away and we had no staff anywhere near it, they didn’t tell us what they did but the temp dropped like a rock, and stayed within (barely) their SLA after that – it was stable after that.

Cabinet Ambient Temperature

There you have it, oh maybe you noticed there’s only one sensor there, yeah the company was that cheap they didn’t want to pay for a second sensor, can you believe that, so glad I’m not there anymore (and oh the horror stories I’ve heard about the place since! what a riot).

Anyways so fast forward to 2012.  Last Friday we had a storage controller fail (no not 3PAR, another lower end HP storage system), with a strange error message, oddly enough the system did not report there was a problem in the web UI (system health “OK”), but one of the controllers was down when you dug into the details.

So we had that controller replaced (yay 4 hour on site support), the next night the second controller failed with the same reason. HP came out again and poked at it, at one point there was a temperature alarm but the on site tech said he thought it was a false alarm, they restarted the controller again and it’s been stable since.

So today I finally had some time to start hooking up the monitoring for the temperature sensors in that facility, it’s a really small deployment, just 1 rack, so 4 sensors.

I was on site a couple of months ago and at the time I sent an email noting that none of the sensors were showing temperatures higher than 78 degrees (even in the rear of the rack).

So imagine my surprise when I looked at the first round of graphs that said 3 of the 4 sensors were now reporting 90 degrees or hotter temperature, and the 4th(near the floor) was reporting 78 degrees.

Wow, that is toasty, freakin hot more like it. So I figured maybe one of the sensors got moved to the rear of the rack, I looked at the switch temperatures and compared them with our other facility, the hotter facility was a few degrees hotter (4C), not a whole lot.

The servers told another story though.

Before I go on let me say that in all cases the hardware reports the systems are “within operating range”, everything says “OK” for temperature – it’s just way above my own comfort zone.

Here is a comparison of two servers at each facility, the server configuration hardware and software is identical, the load in both cases is really low, actually load at the hot facility would probably be less given the time of day (it’s in Europe so after hours). Though in the grand scheme of things I think the load in both cases is so low that it wouldn’t influence temperature much between the two. Ambient temperature is one of 23 temperature sensors on the system.

Data CenterDeviceLocationAmbient Temperature Fan Speeds (0-100%)
[6 fans per server]
Hot Data CenterServer XRoughly 1/3rd from bottom of rack89.6 F90 / 90 / 90 / 78 / 54 / 50
Normal Data CenterServer XRoughly 1/3rd from bottom of rack66.2 F60 / 60 / 57 / 57 / 43 / 40
Hot Data CenterServer YRoughly 1/3rd from bottom of rack87.8 F90 / 90 / 72 / 72 / 50 / 50
Normal Data CenterServer YBottom of Rack66.2 F59 / 59 / 57 / 57 / 43 / 40

That’s a pretty stark contrast, now compare that to some of the external sensor data from the ServerTech PDU temperature probes:

LocationAmbient Temperature (one number per sensor)Relative Humidity (one number per sensor)
Hot Data Center - Rear of Rack95 / 8828 / 23
Normal Data Center - Rear of Rack84 / 84 / 76 / 8044 / 38 / 35 / 33
Hot Data Center - Front of Rack90 / 7942 / 31
Normal Data Center - Front of Rack75 / 70 / 70 / 7058 / 58 / 58 / 47

Again pretty stark contrast. Given that all equipment (even the storage equipment that had issues last week) is in “normal operating range” there would be no alerts or notification, but my own alerts go off when I see temperatures like this.

The on site personnel used a hand held meter and confirmed the inlet temperature on one of the servers was 30C (86 F), the server itself reports 89.6, I am unsure as to the physical location of the sensor in the server but it seems reasonable that an extra 3-4 degrees from the outside of the server to the inside is possible. The data center’s own sensors report roughly 75 degrees in the room itself, though I’m sure that is due to poor sensor placement.

Temperature readout using a hand held meter

I went to the storage array, and looked at it’s sensor readings – the caveat being I don’t know where the sensors are located (trying to find that out now), in any case:

  • Sensor 1 = 111 F
  • Sensor 2 = 104 F
  • Sensor 3 = 100.4 F
  • Sensor 4 = 104 F

Again the array says everything is “OK”,  I can’t really compare to the other site since the storage is totally different(little 3PAR array), but I do know that the cooler data center has a temperature probe directly in front of the 3PAR controller air inlets, and that sensor is reading 70 F. The only temperature sensors I can find on the 3PAR itself are on the physical disks, which range from 91F to 98F, the disk specs say operating temperature from 5-55C (55C = 131F).

So the lesson here is, once again – invest in your own environmental monitoring equipment – don’t rely on the data center to do it for you, and don’t rely on the internal temperature sensors of the various pieces of equipment (because you can’t extract the true ambient temperature and you really need that if your going to tell the facility they are running too hot).

The other lesson is, once you do have such sensors in place, hook them up to some sort of trending tool so you can see when stuff changes.

PDU Temperature Sensor data

The temperature changes in the image above was from when the on site engineer was poking around.

Some sort of irony here the facility that is running hot is a facility that has a high focus on hot/cold isle containment (though the row we are in is not complete so it is not contained right now), they even got upset when I told them to mount some equipment so the airflow would be reversed. They did it anyway of course, that equipment generates such little heat.

In any case there’s tons of evidence that this other data center is operating too hot! Time to get that fixed..

September 5, 2012

Setting expectations for WP8 and BB10

Filed under: General — Tags: — Nate @ 9:54 am

The more I read about Windows Phone 8, and BlackBerry 10 the more I am reminded of WebOS, especially the die hard community around both platforms.

I like WebOS myself, I still use my devices daily, despite my fondness for the platform I was not delusional about it – if HP had released the Pre3 when they were planning on it last year (right about this time last year), it would of been destroyed by the iPhone (I believed this long before I owned the hardware, now that I’ve been using it for the past year my thoughts haven’t changed at all). It would of been just embarrassing to see. Especially after the lackluster performance of the HP Veer, the world’s smallest smart phone at the time (perhaps still is), as far as I know the Veer never received even a single software update post launch which was sad(and there are some bad bugs in it). By contrast the Pre3 received several software updates even though it was never officially launched in the U.S. and had a tiny launch in Europe.

Anyways back on topic, Windows Phone. I have been loosely following the Windows Phone Central site where many of the die hard WP8 fans seem to hang out at for good reason. They were so excited about the launch of the newest Nokia phone, they raged against users who had lost faith in the platform, even raged against manufacturers that seem to be losing faith in the platform.

Microsoft and Nokia have tried to hype up the announcement that came today, and as I’m sure many expected, they over promised and way under delivered. This is the exact same thing HP/Palm was doing (I remember one comment from a HP/Palm person forgot who it was, who said they weren’t going to launch a product that wasn’t perfect – of course the only time they did that was when they shut down the Pre3 before it fully launched).

I feel they failed to truly impress at this event. All the leaks ruined it for me personally. All the new info was boring IMHO

Another user posted

So basically waking up and watching this event was pointless. Nearly everything that was “announced” has already been leaked anyways…seriously, their employees and partners are like swiss cheese when it comes to non-disclosure agreements.

In another article people reacted to the fact that there is no release date, no price, and no firm date for the release of Windows Phone 8 itself. The WP Central site tries to spin the news as positively as it can (much as the Pre Central, oh I’m sorry WebOS Nation site does and did for WebOS). One user wrote

this reveal was choreographed and edited to death.  really a bad demo.  Joe Belfiore just pisses me off the more he comes on stage.  Build 2012 in F-ing november…that’s over a month’s loss of sales opportunities.

I just don’t understand why these other players think announcing (or releasing) products around the time Apple does so is a good idea, it sounded incredibly stupid to me for HP and the Pre3 last year, and it’s even worse this year with Microsoft, Amazon and others trying to steal Apple’s thunder. Samsung did a great job in their latest Galaxy SIII releasing it in June. I have to assume it’s because they have been unable to adjust their product cycles to off set them enough with Apple, or perhaps they just want to try to drive some hype around the holiday season, but if your going up against Apple you really have to bring it. Microsoft/Nokia talk the talk, but they haven’t shown they can hold a candle up to an Apple product launch, so it’s sad to see them even try.

I just saw an interview with the Nokia CEO on CNBC and the only phone he picked up and sort of showed off was not a phone that was announced today, he seemed to focus on their dwindling leadership in the low end phone race.

RIM was sort of saved by further product delays, they did want to launch this fall, but due to problems with the platform they’ve had to postpone the launch yet again to sometime in 2013, more than a year later than the dates I heard originally tossed around a while ago. RIM is busy trying to keep their hype machine primed, offering to essentially bail out (for lack of a better term) developers that make $1,000 or more on applications to the tune of up to $9,000 (for a total of $10,000). If that doesn’t tell you they are bleeding developers like crazy I’m not sure what will. But kudos to them for going the extra mile to try to retain them.

Hopefully, for their sake, RIM can over deliver on their promises, but given how they’ve been for the past year I wouldn’t hold my breath. Nokia seemed to let out another massive disappointment with their announcement today, knecapping Windows Phone 8 before it even gets out of the gate.

One thing Nokia fans can get excited about I suppose is the Nokia touchstone, I mean wireless charging. The Palm wireless charging technology I’ve been using for the past three years is one of the key things I like about the platform. The main downside to it from a mass market perspective from HP/Palm at least is the wireless charging base station was not a cheap accessory, so I suspect many non techies did not opt for it due to the price (which could easily be $50-60 at product launch).

I really would like one of these platforms to do well, trust me I am not a fan of Android nor iOS, it’s just sad to see history repeating itself.

September 2, 2012

Some reasons why Linux on the desktop failed

Filed under: linux — Tags: — Nate @ 10:40 pm

The most recent incarnation of this debate seemed to start with a somewhat interesting article over at Wired who talked to Miguel de Icaza who is a pretty famous desktop developer in Linux, mostly famous for taking what seemed to be controversial stances on implementing Microsoft .NET on Linux in the form of Mono.

And he thinks the real reason Linux lost is that developers started defecting to OS X because the developers behind the toolkits used to build graphical Linux applications didn’t do a good enough job ensuring backward compatibility between different versions of their APIs. “For many years, we broke people’s code,” he says. “OS X did a much better job of ensuring backward compatibility.”

It has since blown up a bit more with lots more people giving their two cents. As a Linux desktop user (who is not a developer) for the past roughly 14 years I think I can speak with some authority based on my own experience. As I think back, I really can’t think of anyone I know personally who has run Linux on the desktop for as long as I have, or more to the point hasn’t tried it and given up on it after not much time had passed – for the most part I can understand why.

For the longest time Linux advocates hoped(myself included) Linux could establish a foot hold as something that was good enough for basic computing tasks, whether it’s web browsing, checking email, basic document writing etc. There are a lot of tools and toys on Linux desktops, most seem to have less function than form(?) at least compared to their commercial counterparts. The iPad took this market opportunity away from Linux — though even without iPad there was no signs that Linux was on the verge of being able to capitalize on that market.

Miguel’s main argument seems to be around backwards compatibility, an argument I raised somewhat recently, backwards compatibility has been really the bane for Linux on the desktop, and for me at least it has been just as much to do with the kernel and other user space stuff as it does the various desktop environments.

Linux on the desktop can work fine if:

  • Your hardware is well supported by your distribution – this will stop you before you get very far at all
  • You can live within the confines of the distribution – if you have any needs that aren’t provided as part of the stock system you are probably in for a world of hurt.

Distributions like Ubuntu, and SuSE before it (honestly not sure what, if anything has replaced Ubuntu today) have made tremendous strides in improving Linux usability from a desktop perspective. Live CDs have helped a lot too, to be able to give the system a test run without ever installing it to your HD is nice.

I suspect most people today don’t remember the days when the installer was entirely text based and you had to fight with XFree86 to figure out the right mode lines for your monitor for X11 to work, and fortunately I don’t think really anyone uses dial up modems anymore so the problems we had back when modems went almost entirely to software in the form of winmodems are no longer an issue. For a while I forked out the cash for Accelerated X, a commercial X11 server that had nice tools and was easy to configure.

The creation of the Common Unix Printer System or CUPS was also a great innovation, printing on Linux before that was honestly almost futile with basic printers, I can’t imagine what it would of been like with more complex printers.

Start at the beginning though – the kernel. The kernel is not, and never really has maintained a stable binary interface for drivers over the years. To the point where I can take a generic driver for say a 2.6.x series kernel and use the same driver (singular driver) on Ubuntu, Red Hat, Gentoo or whatever. I mean you don’t have to look further than how many binary kernel drivers VMware includes with their vmware tools package to see how bad this is, on the version of vmware tools I have on the server that runs this blog there are 197 — yes 197 different kernels supported in there

  • 47 for Ubuntu
  • 55 For Red Hat Enterprise
  • 57 for SuSE Linux Enterprise
  • 39 for various other kernels

In an ideal world I would expect maybe 10 kernels for everything, including kernels that are 64 vs 32 bit.

If none of those kernels work, then yes, vmware does include the source for the drivers and you can build it yourself(provided you have the right development packages installed the process is very easy and fast). But watch out, the next time you upgrade your kernel you may have to repeat the process.

I’ve read in the most recent slashdot discussion where the likes of Alan Cox (haven’t heard his name in years!) said the Linux kernel did have a stable interface as he can run the same code from 1992 on his current system. My response to that is..then why do we have all these issues with drivers.

One of the things that has improved the state of Linux drivers is virtualization – it slashes the amount of driver code needed by probably 99% running the same virtual hardware regardless of the underlying physical hardware. It’s really been nice not to have to fight hardware compatibility recently as a result of this.

There have been times where device makers have released driver disks for Linux, usually for Red Hat based systems, however these often become obsolete fairly quickly. For some things perhaps like video drivers it’s not the end of the world, for the experienced user anyways you still have the ability to install a system and get online and get new drivers.

But if the driver that’s missing is for the storage controller, or perhaps the network card things get more painful.

I’m not trying to complain, I have dealt with these issues for many years and it hasn’t driven me away — but I can totally see how it would drive others away very quickly, and it’s too bad that the folks making the software haven’t put more of an effort into solving this problem.

The answer is usually make it open source – in the form of drivers at least, if the piece of software is widely used then making it open source may be a solution, but I’ve seen time and time again source released and it just rots on the vine because nobody has interest in messing with it (can’t blame them if they don’t need it). If the interface was really stable the driver could probably go unmaintained for several years without needing anyone to look at it (at least through the life of say the 2.6.x kernel)

When it comes to drivers and stuff – for the most part they won’t be released as open source, so don’t get your hopes up. I saw one person say that their company didn’t want to release open source drivers because they feared that they might be in violation of someone else’s patents and releasing the source would make it easier for their competition to be able to determine this.

The kernel driver situation is so bad in my opinion that distributions for the most part don’t back port drivers into their previous releases. Take Ubuntu for example, I run 10.04 LTS on my laptop that I am using now as well as my desktop at work.  I can totally understand if the original released version doesn’t have the latest e1000e (which is totally open source!) driver for my network card at work. But I do not understand that more than a year after it’s release it still doesn’t have this driver. Instead you either have to manage the driver yourself (which I do – nothing new for me), or run a newer version of the distribution (all that for one simple network driver?!). This version of the distribution is supported until April 2013. Please note I am not complaining, I deal with the situation – I’m just stating a fact. This isn’t limited to Ubuntu either, it has applied to every Linux distribution I’ve ever used for the most part. I saw Ubuntu recently updated Skype to the latest version for Linux recently on 10.04 LTS, but they still haven’t budged on that driver (no I haven’t filed a bug/support request, I don’t care enough to do it – I’m simply illustrating a problem that is caused by the lack of a good driver interface in the kernel  – I’m sure this applies to FAR more than just my little e1000e).

People rail on the manufacturers for not releasing source, or not releasing specs. This apparently was pretty common back in the 70s and early 80s. It hasn’t been common in my experience since I have been using computers (going back to about 1990). As more and more things are moved from hardware to software I’m not surprised that companies want to protect this by not releasing source/specs. Many manufacturers have shown they want to support Linux, but if you force them to do so by making them build a hundred different kernel modules for the various systems they aren’t going to put the effort in to doing it. Need to lower the barrier of entry to get more support.

I can understand where the developers are coming from though, they don’t have incentive to make the interfaces backwards and forwards compatible since that does involve quite a bit more work(much of it boring), instead prefer to just break things as the software evolves. I had been hoping that as the systems matured more this would become less common place, but it seems that hasn’t been the case.

So I don’t blame the developers…

But I also don’t blame people for not using Linux on the desktop.

Linux would of come quite a bit further if there was a common way to install drivers for everything from network cards to storage controllers, to printers, video cards, whatever, and have these drivers work across kernel versions, even across minor distribution upgrades. This has never been the case though (I also don’t see anything on the horizon, I don’t see this changing in the next 5 years if it changes ever).

The other issue with Linux is working within the confines of the distribution, this is similar to the kernel driver problem – because different distros are almost always more than a distribution with the same software, the underlying libraries are often incompatible between distributions so a binary on one, especially a complex one that is say a KDE or Gnome application won’t work on another, there are exceptions like Firefox, Chrome etc – though other than perhaps static linking in some cases not sure what they do that other folks can’t do. So the amount of work to support Linux from a desktop perspective is really high. I’ve never minded static linking, to me it’s a small price to pay to improve compatibility with the current situation. Sure you may end up loading multiple copies of the libraries into memory(maybe you haven’t heard but it’s not uncommon to get 4-8GB on a computer these days), sure if there is a security update you have to update the applications that have these older libraries as well. It sucks I suppose, but from my perspective it sucks a lot less than what we have now.  Servers – are an entirely different beast, run by(hopefully) experienced people who can handle this situation better.

BSD folks like to tout their compatibility – though I don’t think that is a fair comparison, comparing two different versions of FreeBSD against Red Hat vs Debian are not fair, comparing two different versions of Red Hat against each other with two different versions of FreeBSD (or NetBSD or OpenBSD or DragonFly BSD …etc)  is more fair. I haven’t tried BSD on the desktop since FreeBSD 4.x, for various reasons it did not give me any reasons to continue using it as a desktop and I haven’t had any reasons to consider it since.

I do like Linux on my desktop, I ran early versions of KDE (pre 1.0), up until around KDE 2, then switched to AfterStep for a while, eventually switching over to Gnome with Ubuntu 7 or 8, I forget. With the addition of an application called Brightside, GNOME 2.x works really well for me. Though for whatever reason I have to launch Brightside manually each time I login, setting it to run automatically on login results in it not working.

I also do like Linux on my servers, I haven’t compiled a kernel from scratch since the 2.2 days, but have been quite comfortable working with the issues operating Linux on the server end, the biggest headaches were always drivers with new hardware, though thankfully with virtualization things are much better now.

The most recent issue I’ve had with Linux on servers has been some combination of Ubuntu 10.04, with LVM and ext4 along with enterprise storage. Under heavy I/O have have seen many times ext4 come to a grinding halt. I have read that Red Hat explicitly requires that barriers be disabled with ext4 on enterprise storage, though that hasn’t helped me. My only working solution has been to switch back to ext3 (which for me is not an issue). The symtoms are very high system cpu usage, little to no i/o (really any attempts to do I/O result in the attempt freezing up), and when I turn on kernel debugging it seems the system is flooded with ext4 messages. Nothing short of a complete power cycle can recover the system in that state. Fortunately all of my root volumes are ext3 so it doesn’t prevent someone from logging in and poking around. I’ve looked high and low and have not found any answers. I had never seen this issue on ext3, and the past 9 months has been the first time I have run ext4 on enterprise storage. Maybe a bug specific to Ubuntu, am not sure. LVM is vital when maximizing utilization using thin provisioning in my experience, so I’m not about to stop using LVM, as much as 3PAR’s marketing material may say you can get rid of your volume managers – don’t.

 

August 27, 2012

vSphere 5.1: what 5.0 should of been!

Filed under: General — Tags: — Nate @ 6:40 pm

I’m not going to bore you with all the mundane details about what is new, so many other folks are doing that, but here is a post from VMware which has links to pdfs as to what is new.

It looks pretty decent, the licensing change is welcome, though not everyone agrees with that. I find it interesting that the web console is going to be the main GUI to manage vSphere going forward. I found the web console in 5.0 severely lacking, but I’m sure it’s improved in 5.1. Anyone happen to know if the new console is backwards compatible with vCenter 5.0 ? Also I wonder if this web console applies to managing ESXi hosts directly (without vCenter)? I assume it doesn’t apply(yet) ?

I don’t see myself upgrading anytime before the end of the year, but it does seem strongly to me that this 5.1 release is what 5.0 should of been last year.

I find this race to a million iops quite silly, whether it is VMware’s latest claim of 1 million iops in a VM, or EMC’s latest claim, or HDS’s latest claim, everyone is trying to show they can do a million too, and the fine print always seems to point to a 100% read workload, maybe customers will buy their arrays with their data pre-loaded, so they don’t have to do any writes to them.

 

August 24, 2012

3PAR: Helping and Hurting HP ?

Filed under: Storage — Tags: — Nate @ 8:59 am

Here I go, another blog post starting with question. Yet another post sparking speculation on my part due to an article by our friends at The Register who were kind enough to do some number crunching of HP’s latest quarterly numbers were storage revenues were down about 5%.

Apparently a big chunk of the downward slide for revenues was declines in EVA and tape, offset to some degree by 3PAR and StoreOnce de-dupe products.

I suppose this thought could apply to both scenarios, but I’ll focus on the disk end, since I have no background on StoreOnce.

Before HP acquired 3PAR, obviously EVA was a juicy target to go after to replace EVAs with 3PARs. The pitch was certainly you can get a hell of a lot more done on less 3PAR than you can with more EVA. So you’ll end up saving money. I’ve never used EVA before myself, heard some good aspects of it and some really bad aspects of it, I don’t think I’d ever want to use EVA regardless.

I am sure that 3PAR reps (those that haven’t left anyways – I’ve heard from numerous sources they outclass their HP counterparts by leagues and leagues), who are now responsible for pitching HP’s entire portfolio obvious have a strong existing bias towards 3PAR and away from the other HP products. They try to keep a balanced viewpoint but I’m sure that’s hard to do, especially after they’ve been spending so much time telling the world how much these other products are bad and why the customer should use 3PAR instead. Can’t blame them, its a tough switch to make.

So, assuming you can get a hell of a lot more done on a smaller/fewer 3PAR system(s) than EVA  – which I think is totally true, (with some caveat as to what sort of discounts some may be able to score on EVA, 3PAR has traditionally had some strict margin rules where they have no problem walking away from a deal if the margin is too low), add to that the general bias of at least part of the sales force, as well as HP’s general promotion that 3PAR is the future, and you can quite possibly get lower overall revenue while the customers are saving money by having to buy fewer array resources to accomplish the same (or more) tasks.

3PAR revenue was up more than 60% apparently, on top of the previous gains made since the acquisition.

It would be very interesting to me to see how much consolidation some of these deals end up being – traditionally NetApp I think has been the easiest target for 3PAR, I’ve seen some absolutely massive consolidation done in the past with those products, it was almost comical in some cases. I bet EVA is similar.

Now the downside to the lower revenues, and I’ve seen this at both Dell and HP – both companies are feeling tremendous pressure to try to outperform, they haven’t been able to do it on the revenue side, so they’ve been squeezing on internal costs, which really can degrade services. Overall quality of the sales forces at the likes of HP and Dell have traditionally been terrible, compared to the smaller company counterparts (at least in storage). Add to that the internal politics and region limitations that the companies place on their sales forces further complicates and frustrates the quality people internally as well as customers externally. Myself I was unable to get anything out of a local HP/3PAR account team for months in the Bay Area, so I reached out to my friends in Seattle and they turned some stuff around for me in a matter of hours no questions asked, and they didn’t get any credit (from HP) for it either. Really sad situation for both sides.

I don’t have much hope that HP will be willing or able to retain the top quality 3PAR folks at least on the sales side over the medium term, they, like Dell seem focused on driving down costs rather than keeping quality high, which is a double edged sword. The back end folks will probably stick around for longer, given that 3PAR is one of the crown jewels in HP’s enterprise portfolio.

For some reason I’m immediately reminded of this quote from Office Space:

[..] "that is not right, Michael. For five years now, you've worked your ass  off at Initech, hoping for a promotion or some kind of profit sharing  or something. Five years of your mid-20s now, gone. And you're gonna go  in tomorrow and they're gonna throw you out into the street. You know  why? So Bill Lumbergh's stock will go up a quarter of a point."

 

One of 3PAR’s weak points has been at the low end of the market, say sub $100k deals, is a space 3PAR has never tried to compete in.  Apparently according to The Register the HP P4000/Lefthand side of things is not doing so hot, and also seemed to be HP’s go-to product for this price range. This product range is what HP used to be excited about, before 3PAR, I attended a storage briefing at a VMware User group meeting just before HP bought 3PAR, expecting some sort of broad storage overview, but it was entirely Lefthand focused. While Lefthand has some interesting tech (the network RAID is pretty neat), for the most part I’d rather pay more and use 3PAR obviously.

I wonder what will happen to Lefthand in the future, will the best of it’s tech get rolled up into 3PAR? or vise versa? Or maybe it will just stay where it’s at, the one good thing Lefthand has is the VSA, it’s not as complete as I’d like to see it, but it’s one of the very few VSAs out there.

Dell has been busy trying to integrate their various storage acquisitions whether it’s Compellent, Ocarnia, Exanet, and I think there was one or two more that I don’t remember. Storage revenues there down as well. I’m not sure how much of the decline has to do with Dell terminating the EMC reselling stuff at this point, but it seems like a likely contributor to the declines in their case.

August 23, 2012

Real time storage auto tiering?

Filed under: Storage — Tags: — Nate @ 9:42 am

Was reading this article over by our friends at The Register, and apparently there is some new system from DotHill that claims to provide real time storage tiering – that is moving data between tiers every 5 seconds.

The AssuredSAN Pro 5000 has a pretty bad name, borrows the autonomic term that 3PAR loves to use so much(I was never too fond of it, preferred the more generic term automatic), in describing their real time tiering. According to their PDF, they move data around in 4MB increments, far below most other array makers.

From the PDF –

  • Scoring to maintain a current page ranking on each and every I/O using an efficient process that adds less than one microsecond of overhead. The algorithm takes into account both the frequency and recency of access. For example, a page that has been accessed 5 times in the last 100 seconds would get a high score.
  •  Scanning for all high-scoring pages occurs every 5 seconds, utilizing less than 1.0% of the system’s CPU. Those pages with the highest scores then become candidates for promotion to the higher-performing SSD tier.
  •  Sorting is the process that actually moves or migrates the pages: high scoring pages from HDD to SSD; low scoring pages from SDD back to HDD. Less than 80 MB of data are moved during any 5 second sort to have minimal impact on overall system performance.

80MB of data every 5 seconds is quite a bit for a small storage system, I have heard of situations where auto tiering has such an impact that it actually made things worse due to so much data moving around internally on the system, and had to be disabled. I would hope they have some other safe gaurds in there like watching spindle latency and scheduling the movements to be low priority, and perhaps even canceling movements if the job can’t be completed in some period of time.

The long delay time between data movements is perhaps the #1 reason why I have yet to buy into the whole automagic storage tiering concept. My primary source of storage bottlenecks over the years has been some out-of-nowhere job that generates a massive amount of writes and blows out the caches on the system. Sometimes the number of I/Os inbound is really small too, but the I/O size can be really big (5-10x normal) and is then split up into smaller I/Os on the back end and the IOPS are multiplied as a result. The worst offender was some special application I supported a few years ago which would, as part of it’s daily process dump 10s of GB of data from dozens of servers in parallel to the storage system as fast as it could. This didn’t end well as you can probably imagine, at the peak we had about roughly 60GB of RAM cache between the SAN and NAS controllers. I tried to get them to re-architect the app to use something like local Fusion IO storage but they did not they needed shared NFS. I suspect this sort of process would not be helped too much by automatic storage tiering because I’m sure the blocks are changing each day to some degree. This is also why I have not been a fan of things like SSD read caches (hey there NetApp!), and of course having a SSD-accelerated write cache on the server end doesn’t make a lot of sense either since you could lose data in the event the server fails. Unless you have some sort of fancy mirroring system, to mirror the writes to another server but that sounds complicated and problematic I suspect.

Compellent does have one type of tiering that is real time though – all writes by default go to the highest tier and then are moved down to lower tiers later. This particular tiering is even included in the base software license. It’s a feature I wish 3PAR had.

This DotHill system also supports thin provisioning, though no mention of thin reclaiming, not too surprising given the market this is aimed at.

They also claim to have some sort of rapid rebuild, by striping volumes over multiple RAID sets, I suppose this is less common in the markets they serve (certainly isn’t possible on some of the DotHill models), this of course has been the norm for a decade or more on larger systems. Rapid rebuild to me obviously involves sub disk distributed RAID.

Given that DotHill is a brand that others frequently OEM I wonder if this particular tech will bubble up under another name, or will the bigger names pass on this in the fear it may eat into their own storage system sales.

August 22, 2012

Are people really waiting for Windows 8?

Filed under: General — Tags: — Nate @ 11:04 am

UPDATED  – Dell came out with their quarterly results and their PC business wasn’t doing well, down 22% from a year ago.

A common theme I see popping up are people claiming folks are holding off until Windows 8 before buying their systems. Do you think this is true? I can imagine some folks holding off to buy the MS Tablet or something, but I can’t imagine many people (I’d bet it’s a rounding error) waiting for the desktop/laptop version of Windows 8 before buying their new PC.  Especially given apparently 70% of Dells PC sales go to commercial/government with only 30% going to consumer.

UPDATE – HP released their results and their PC shipments were similarly poor with revenues down 10-12%.

They’d rather not admit that a weak economy combined with the rise of tablets not running Microsoft software is probably where most of the blame lies. A market that Linux advocates long hoped to try to capture by being “good enough” to do things like browse the web, read email etc, but for many reasons was never able to really establish any foot hold. I remember my grandfather had such a Linux system, forgot who made it, but it ran entirely off of a CD, with perhaps only a few megs of flash storage (this was about 10-12 years ago). It was slow but it mostly worked, he certainly got far fewer viruses on that then when he eventually went to windows a few years later(Dell had to re-install his system several times a year due to malware infestations – he refused to stop going to dodgy porn sites).

What’s new and good in Windows 8 vs 7 that would have folks want to hold back on a new desktop or laptop? It’s pitched as the biggest product launch since Windows 95 and I believe that, though I don’t see anywhere near the leap in technology from Windows 7 to 8 that happened from Win3.x to 95. (sort of like my feelings on vSphere 3.5->4 vs 4->5).

I suspect it’s just an excuse, time will tell if there is a massive surge in PC buying shortly after Windows 8 launches but I don’t expect there to be. The new Metro (or whatever they are calling it) ecosystem is minuscule compared to the existing windows ecosystem. Hell, all other ecosystems pale in comparison to the current windows ecosystem.

Do you know if people are holding back waiting for Windows 8 on a desktop or laptop style device? If so I’d be curious to hear the reasons.

I fear for Microsoft that Windows 8 will hit the ground with a thud. It’ll sell millions just because there is such a massive install base of Windows folks (Vista sold quite a bit too remember that!).  Unlike some other players (*cough* HP WebOS *cough*), even if it is a thud initially Microsoft won’t give up. I recall similar hype around Windows phone 7 and that hit with a thud as well and has gone nowhere. In short – MS is setting the expectations too high.

Something I did learn recently which I was not aware of before, one of my friends at Microsoft mentioned that the Windows Phone 7 platform was mostly acquired from another company(forgot which), Microsoft then went and gutted it and the result is what we have today. He had long lost all faith in Microsoft, in their ability to execute, stifling projects that have good prospects while promoting others that have none. I suppose that sort of thing is typical for a really big company. I don’t know how he (or others) can put up with it without going crazy. He didn’t have much positive things to say about Windows phone, nor did his girlfriend who also works at Microsoft. It was sort of “meh”.

They’ll keep trying though, Microsoft that is, throw enough money at anything and you’ll eventually get it right, it may cost them a few more billion, and a few more years, but it’s a big important market, so it’s certainly a worthwhile investment.

I do find it funny while Ballmer was out trying to destroy Google, Apple comes out of nowhere and takes the torch and runs with it, and it took Microsoft many years to regroup and try to respond.

I don’t hate Microsoft, haven’t for a while, I do sort of feel sorry for them though, their massive fall from the top of the world to where they are now. They still have tons of cash mind you.. but from pretty much every other angle they aren’t doing so well.

August 21, 2012

When is a million a million?

Filed under: Storage — Tags: — Nate @ 7:24 pm

I was doing my usual reading The Register, specifically this article and something popped into my mind and I wanted to write a brief note about it.

I came across the SPC-1 results for the Huawei OceanStor Dorado 5100 before I saw the article and didn’t think a whole lot about it.

I got curious when I read the news article though so I did some quick math – the Dorado 5100 is powered by 96 x 200GB SSDs and 96GB of cache in a dual controller active-active configuration. Putting out an impressive 600,000 IOPS with the lowest latency (by far) that I have seen anyways. Also they did have a somewhat reasonable unused storage ratio of 32.35% (I would of liked to have seen much better given  the performance of the box but I’ll take what I can get).

But the numbers aren’t too surprising – I mean SSDs are really fast right. What got me curious though is the # of IOPS coming out of each SSD  to the front end, in this case it comes to 6,250 IOPS/SSD. Compared to some of the fastest disk-based systems this is about 25x faster per disk than spinning rust. There is no indication that I can see at least that tells what specific sort of SSD technology they are using(other than SLC). But 6,250 per disk seems like a far cry from the 10s of thousands of IOPS many SSDs claim to be able to do.

I’m not trying to say it’s bad or anything but I found the stat curious.

I went ahead and looked at another all-SSD solution the IBM V7000, this time 18 x 200GB SSDs are providing roughly 120,000 IOPS also with really good latency with 16GB of data cache between the pair of controllers.  Once again the numbers come to roughly 6,600 IOPS per SSD. IBM ran at an even better unused storage ratio of just under 15%, hard to get much better than that.

Texas memory systems (recently acquired by IBM), posted results for their RamSan-630 about a year ago, with 20 x 640GB SSDs pushing out roughly 400,000 IOPS with pretty good latency. This time however the numbers change – around 20,000 IOPS per SSD here, as far as I can tell there is no RAM cache either. The TMS system came in at a 20% unused storage ratio.

While there are no official results, HP did announce not long ago that an ‘all SSD’ variant of the P10000(just realized it is kind of strange to have two sub models(V400 and V800 which were the original 3PAR models) of the larger P10000 model), which said would get the same 450,000 IOPS on 512 x SSDs. The difference here is pretty stark with each SSD theoretically putting out only 878 IOPS(so roughly 3.5x faster than spinning rust).

At least originally I know originally 3PAR chose a slower STEC Mach8IOPS SSD primarily due to cost (it was something like 60% cheaper). STEC’s own website shows the same SSD getting 10,000 IOPS (on a read test – whereas the disk they compared it to seemed to give around 250 IOPS).  Still though  you can tap out the 8 controllers with almost 1/4th the number of disks supported with these SSDs. I don’t know whether or not the current generation of systems uses the same SSD or not.

I’ll be the first to admit an all-SSD P10000 doesn’t make a lot of sense to me, though it’s nice that customers have that option if that’s what they want (I never understood why all-SSD was not available before that didn’t make sense either). HP says it is 70% less expensive than an all-disk variant, though are not specific whether they are using 100GB SSDs (I assume they are) vs 200GB SSDs.

Both TMS and Huawei advertise their respective systems as being “1 million IOPS”, I suppose if you took one of each and striped them together that’s about what you’d get ! Sort of reminds me of a slide show presentation I got from Hitachi right before their AMS2000-series launched one of the slides showed the # of IOPS from cache (they did not have a number for IOPS from disk at the time), which didn’t seem like a terribly useful statistic.

So here you have individual SSDs providing anywhere from 900 to 20,000 IOPS per disk on the same test…

I’d really love to see SPC-1 results for the likes of Pure Storage, Nimble Storage, Nimbus Storage, and perhaps even someone like Tintri, just to see how they measure up on a common playing field, with a non trivial utilization rate. Especially with claims like this from Nimbus saying they can do 20M IOPS/rack, does that mean at 10% of usable capacity or greater than 50%? I really can’t imagine what sort of workload would need that kind of I/O but there’s probably one or two folks out there that can leverage it.

We now take you back to your regularly scheduled programming..

« Newer Posts — Older Posts »

Powered by WordPress