Diggin' technology every day


HP Discover 2014 – Las Vegas

TechOps Guy: Nate

HP Discover 2014I'm going to be attending my first HP Discover in two weeks in Las Vegas. HP has asked me for a while to go but I do not like big trade shows(or anywhere with large crowds of people), so until now have shied away.

I had a really good time at the HP Storage tech day and Nth symposium last year so I decided I wanted to try out Discover this year given that I know at least some folks that will be there and we'll be in a somewhat organized group of "bloggers" led by Calvin Zito the HP Storage blogger.

I've never been to Las Vegas before but I'll be there from June 8th and leaving on the 13th. After that I'm going to Arizona to check out the Grand Canyon and a few other places for a few days and return home the following week some time.

Looking forward to meeting some folks there, should be pretty fun.


Back from Amsterdam

TechOps Guy: Nate

I'm back from Amsterdam - it was about what I expected. I basically stuck to the hotel and the data center - I even skipped out on that little cruise I pre paid for, just didn't feel like going. I knew I disliked traveling and this trip was a massive reminder as to why. About the only thing that was a positive surprise for me was the long haul flights. I was dreading it at first but the nice reclining seats and big screen LCDs allowed me to kick back and stretch my legs without getting the usual cramps and discomfort. My flight to Amsterdam was on a single airline, made a stop in Chicago where the transfer of plans was amazingly short - it was about 150 feet between the gates I was afraid that it was going to be far and maybe I'd miss the flight (I don't have much recent flying experience the last time I had to make a connecting flight was I'd wager 20 years ago).

I got confused as to my flight schedule(wasn't going to be the first time) and I arrived in Amsterdam about eight hours before I thought I was going to arrive. The hotel was alright, I mean for the price at least, it was around $200/night or something which seemed pretty typical for a city room. First thing I noticed is it took me a good 3-5 minutes to figure out how the lights worked (had to put hotel key card in a slot to activate them). Took a shower after the long flight - no washcloth ? Maybe it is not typical in Europe I don't know, I seem to recall washcloths at hotels I was at in Asia growing up. The toilet was a very strange design, it was like this, which had a couple drawbacks. The mini bar in the room was automatic, I didn't notice that until the 2nd day, so you can't even take something out to look at it without being charged. I ended up taking quite a bit of things out. There was a sort of mini mart at a shell station about a half mile away that I walked to to buy drinks and stuff on a couple occasions the selection paled in comparison to similar stores in the U.S. The first time I went I literally saw a line of cars at the pumps. I don't know if gas was cheap or if it was a rush hour or the only gas station in the area but  it really reminded me of seeing the pictures of the gas shortages in the 70s in the U.S. There wasn't many pumps - I think 4 or 5, I'd say less than half the typical gas station here.

On the first leg of my flight the passenger next to me said watch out for the bikes - but didn't elaborate. Wow - I had not seen so many bikes since I lived in China in 89-90. They certainly have their bike infrastructure laid out pretty well with dedicated pathways for bikes as well as dedicated street crossing signals etc. On one of my walks around the hotel area I walked through what appeared to be their version of the Park and Ride. Where here the park and rides are filled with cars and parking lots, this one was filled with bikes and was pretty much entirely under a freeway overpass. It seemed like a large number of bikes weren't even locked up. The overall quality of the bikes seemed low I suppose that is at least partly to reduce theft by not having nice fancy bikes I'm not sure. More than anything when I saw the bike stuff it made me think this must be what those hippies in Seattle and SFO want. It was certainly an interesting design, too much of a culture shock for me though.

I found the intersections very confusing and am even more glad I did not try to rent a car while I was there.

Speaking of cars, wow are they small over there, I struggle to think of seeing even a single pickup truck (of any size) while I was there. I saw a bunch of cars like mine, and there was this other really tiny car, which made those tiny Smart cars look big, it was smaller than a golf cart. I missed a few opportunities to take pictures of them, I'm sure I could find them online somewhere. The taxi drivers drove sort of crazy, drifting between lanes and stuff, one of them blew way through a red light(the other lights must've turned green already) which was freaky. I recall on that same trip we were behind some kind of small van that had a radiation warning sign on it.

The data center was -- interesting I guess. Everyone had to wear protective booties around their shoes while on the floor which was a first for me, I think way over kill. Nothing really exciting here, I got everything done that I needed to get done.

I spent hours looking online for places to go but could not find anything that I was interested in. Well there was one thing I just couldn't figure out how to do it. I was really interested in seeing the big water structures they used to hold back the water. The biggest of them appeared to be a 2 hour drive away from the city ( too far). There was a couple tours that hit them but they were minimum 8 hour commitment which was too long. This is my first trip where I did not have a car at the destination and that was a good reason why I didn't do anything or go anywhere, normally I would just roam around but relying on taxis I really had to have a precise destination. I wasn't about to rent a car, I really did not feel anywhere comfortable enough to drive in a foreign country like that. While everyone said "they all speak english!", most people did speak great english, but the destinations for me for the most part were unpronounceable and not understandable (Schepenbergweg was the street the data center was on - I heard it pronounced at least a dozen times and at the end was no closer to beginning to pronounce it myself than hearing it the first time). Because of the $20 per megabyte roaming data fees on my phone I kept the data services on the phone disabled throughout my trip there which of course limited my ability to find stuff while not at the hotel or data center. I was especially worried of getting lost and having to call for a taxi and not be able to pronounce where I was and the taxi not being able to find me. I don't know how it was like in the real down town parts of town but in all the places I visited while growing up in Asia there was taxis everywhere you could just flag down and get one. I did not see this situation in the areas I was at in Amsterdam. The hotel called me a taxi to go to the data center and I asked the security guards at the data center to call me a taxi to get back.

So in the end I ate most of my meals at the hotel, never went to the down town part of town, I walked around a bit around the hotel and took some pictures of the area, nothing special. It really reminded me how much I dislike traveling in general.

The flight back was a little more frustrating, having to stop in London and go through customs and immigration and a pretty long trip to change terminals, it seems like I barely made the flight despite having a 2 hour stop over. I had to ask multiple people for help while there too because while I had a boarding pass it didn't tell me which gate or even which terminal to go to. Even once I knew where to go, getting there wasn't clear either. The whole place was very confusing, and as a result very frustrating.

This is the first trip I've taken in recent memory where I was really excited about going home. I wasn't looking forward to it to begin with and it turned out about the way I expected. Hopefully that's my last trip for a long time to come.

I thought about going somewhere fancy to eat or something, but I really couldn't find anything of interest. Add to that I don't like going out alone, if I'm with a friend things are different. When it comes to things like fancy steak or pasta or whatever I really don't have the sensitivity to tell the difference between most of them so I wouldn't be able to appreciate the good stuff so there really isn't a whole lot of point of me going. There was a BBQ + Grill near the data center (emphasis on was), the sign was still up but the building was empty. I went to two different nice places with a local friend when I was in Atlanta that I really enjoyed, I tried finding something sort of along those lines in Amsterdam but came up with nothing. Most of the places seemed too exotic or too fancy/upper class.

Apparently I left on the day things were going to get crazy, some special soccer game was being played on Saturday afternoon (I left at around noon). I've never been much of a soccer fan at least not since I played it back in 5th grade and earlier years. About the only sport I do enjoy watching is pro football, and even then my interest has been waning over the recent years.

I did all of my shopping at the Airport, picked up a bunch of dutch chocolate going to give most of it away, I tried some of it and it tastes like regular chocolate. I live a mile or so away from a pretty big Sees Candy operation, I bought some of their stuff for Christmas gifts last year, it tastes similar to the Dutch stuff if not better. Picked up a couple picture books of the area, along with some shot glasses for friends and/or family or something.

I got back a full day earlier than I expected. I was absolutely sure yesterday was Monday when I woke up at 5:30AM and turned to CNBC only to see it was Sunday. I got back on Saturday afternoon.

Contrast that with my next trip, which I think will be early July at this point, road trip up to Seattle. I decided to take the coast up north at least to Crescent City, CA. I've been wanting to take my new car along the coast since I bought it over a year ago. I made the coastal trip a couple of times several years ago but not in a car as fun to drive as the one I have at the moment. I'm not sure if I will spend two or three days driving up. I'm really looking forward to that. I think it may of been really cool to go along the coast of the Netherlands but I really didn't have a way to make that happen while I was there.

One of my friends from SEA is in town for a few days I intend to take tomorrow off and go see him down in Morgan Hill, CA (60 miles away), should be good times to catch up and hang out at this nice place he is talking about.

Tagged as: Comments Off

VMware increases core counts in 4.1 licensing

TechOps Guy: Nate

I just came across this mention on AMD's blog. They note that vSphere 4.1 Update 2 included a CPU licensing change -

For the AMD Opteron 6200 and 4200 series (Family 15h) processors, ESX/ESXi 4.1 Update 2 treats each core within a compute unit as an independent core, except while applying licenses. For the purpose of licensing, ESX/ESXi treats each compute unit as a core. For example, a processor with 8 compute units can provide the processor equivalent of 16 cores on ESX/ESXi 4.1 Update 2. However, ESX/ESXi 4.1 Update 2 only requires an 8 core license for each 16-core processor.

I had not heard of that before, so it's news to me! So not only is the physical cost of the Opteron 6200 cheaper than the 6100, the licensing cost is half as much (per core). AMD's blog post above shows some pretty impressive results where a pair of quad socket 6200 blades outperforming a pair of quad socket 10-core Intel blades(2 sockets populated per blade) and at the same time the 6200 solution costs half as much (per VM). Though it's also comparing vSphere 4.1 vs 5.0, since the Opteron 6200 results seem to be the first vSphere 5.0 VMmark results posted. Also the Intel solution has twice the ram as the Opteron but still loses out.

Based on what I see it seems VMmark is more CPU bound than memory(capacity bound), which I suppose I can understand but still in the vast majority of situations the systems are not CPU bound. People tend to load up more on CPUs so they can get more memory capacity. I won't have real numbers for probably two months but I'm expecting CPU usage on this new cluster I am building to be at least half the amount of memory usage.

The change sounds Oracle-esque in licensing where they have fairly complicated decisions they made to determine how many "Oracle cores" you have on your physical processor.

I am traveling tonight to Atlanta to deploy a new vSphere cluster with Opteron 6100s, I was going to go with vSphere 5 because of the license limits on vSphere 4.1 not supporting 16 core processors. Now I see 4.1 does support it so I have about 48 hours to think about whether or not I want to change my mind. I do like vSphere 5's inclusion of LLDP support, more vCPUs per VM. Though really even now after I have been looking through what is in vSphere 5 I don't see anything game changing, nothing remotely, in my opinion like the change to vSphere 4.0 from ESX 3.5.

Weigh the benefits of what's new in vSphere 5 vs having the ability to have unlimited memory(well, up to 1TB, which for me is unlimited from a practical standpoint) in my hosts for no additional licensing cost...

I'm already licensed for vSphere 5 since we bought it after the deadline of the end of September.

Mad props to AMD for getting VMware to tweak their licensing.

Decisions, decisions..


Wired or Wireless?

TechOps Guy: Nate

I'll start out by saying I've never been a fan of Wifi, it's always felt like a nice gimmick-like feature to have but other than that I usually steered clear. Wifi has been deployed at all companies I have worked at in the past 7-8 years though in all cases I was never responsible for that (I haven't done internal IT since 2002, at which time wifi was still in it's early stages(assuming it was out at all yet? I don't remember) and was not deployed widely at all - including at my company). I could probably count on one hand the number of public wifi networks I have used over the years, excluding hotels (of which there was probably ten).

In the early days it was mostly because of paranoia around security/encryption though over the past several years encryption has really picked up and helped that area a lot. There is still a little bit of fear in me that the encryption is not up to snuff, and I would prefer using a VPN on top of wifi to make it even more secure, only really then would I feel comfortable from a security standpoint of using wifi.

From a security standpoint I am less concerned about people intercepting my transmissions over wifi than I am about people breaking into my home network over wifi (which usually happens by intercepting transmissions - my point is more of the content of what I'm transferring, if it is important is always protected by SSL or SSH or in the case of communicating with my colo or cloud hosted server there is a OpenVPN SSL layer under that as well).

Many years ago, I want to say 2005-2006 time frame, there was quite a bit of hype around the Linksys WRT-54G wifi router, for being easy to replace the firmware with custom stuff and get more functionality out of it. So I ordered one at the time, put dd-wrt on it, which is a custom firmware that was talked a lot about back then (is there something better out there? I haven't looked). I never ended up hooking it to my home network, just a crossover cable to my laptop to look at the features.

Then I put it back in it's box and put it in storage.

Until earlier this week, when I decided to break it out again to play with in combination with my new HP Touchpad, which can only talk over Wifi.

My first few days with the Touchpad involved having it use my Sprint 3G/4G Mifi access point. As I mentioned earlier I don't care about people seeing my wifi transmissions I care about protecting my home network. Since the Mifi is not even remotely related to my home network I had no problem using it for extended periods.

The problem with the Mifi, from my apartment is the performance. At best I can get 20% signal strength for 4G, and I can get maybe 80% signal strength for 3G, latency is quite bad in both cases, and throughput isn't the best either, a lot of times it felt like I was on a 56k modem. Other times it was faster. For the most part I used 3G because it was more reliable for my location, however I do have a 5 gig data cap/month for 3G so considering I started using the Touchpad on the 1st of the month I got kind of concerned I may run into that playing with the new toy during the first month. I just checked Sprint's site and I don't see a way to see intra month data usage, only data usage for the month once it's completed. The mifi tracks data usage while it is running but this data is not persisted across reboots, and I think it's also reset if the mifi changes between 3G and 4G services. I have unlimited 4G data, but the signal strength where I'm at just isn't strong enough.

I looked into the possibility of replacing my Mifi with newer technology, but after reading some customer reviews of the newer stuff it seemed unlikely I would get a significant improvement in performance at my location, enough to justify the cost of the upgrade at least so I decided against that for now.

So I broke out the WRT-54G access point and hooked it up. Installed the latest recommended version of firmware, configured the thing and hooked up the touchpad.

I knew there was a pretty high number of personal access points deployed near me, it was not uncommon to see more than 20 SSIDs being broadcast at any given time. So interference was going to be an issue. At one point my laptop showed me that 42 access points were broadcasting SSIDs. And that of course does not even count the ones that are not broadcasting, who knows how many there are there, I haven't tried to get that number.

With my laptop and touchpad being located no more than 5 feet away from the AP, I had signal strengths of roughly 65-75%. To me that seemed really low given the proximity. I suspected significant interference was causing signal loss. Only when I put the touchpad within say 10 inches of the antenna from the AP did the signal strength go above 90%.


Looking into the large number of receive errors told me that those errors are caused almost entirely by interference.

So then I wanted to see what channels were most being used and try to use a channel that has less congestion, the AP defaulted to channel 6.

The last time I mucked with wifi on linux there seemed to be an endless stream of wireless scanning, cracking, hacking tools. Much to my shock and surprise these days most of those tools haven't been maintained in 5-6-7-8+ years. There aren't many left. Sadly enough the default Ubuntu wifi apps do not report channels they just report SSIDs. So I went on a quest to find a tool I could use. I finally came across something called wifi radar, which did the job more or less.

I counted about 25 broadcasting SSIDs using wifi radar, nearly half of them if I recall right were on channel 6. A bunch more on 11 and 1, the other two major channels. My WRT54G had channels going all the way up to 14. I recall reading several years ago about frequency restrictions in different places, but in any case I tried channel 14 (which is banned in the US). Wifi router said it was channel 14, but neither my laptop nor Touchpad would connect. I suspect since they flat out don't support it. No big deal.

Then I went to channel 13. Laptop immediately connected, Touchpad did not. Channel 13 is banned in many areas, but is allowed in the U.S. if the power level is low.

Next I went to channel 12. Laptop immediately connected again, Touchpad did not. This time I got suspicious of the Touchpad. So I fired up my Palm Pre, which uses an older version of the same operating system. It saw my wifi router on channel 12 no problem. But the Touchpad remained unable to connect even if I manually input the SSID. Channel 12 is also allowed in the U.S. if the power level is low enough.

So I ended up on channel 11. Everything could see everything at that point. I enabled WPA2 encryption, enabled MAC address filtering (yes I know you can spoof MACs pretty easily on wifi, but at the same time I have only 2 devices I'll ever connect so blah). I don't have a functional VPN yet mainly because I don't have a way (yet) to access VPN on the Touchpad, it has built in support for two types of Cisco VPNs but that's it. I installed OpenVPN on it but I have no way to launch it on demand without being connected to the USB terminal.  I suppose I could just leave it running and in theory it should automatically connect when it finds a network but I haven't tried that.

So on to my last point on wifi - interference. As I mentioned earlier signal quality was not good even being a few feet away from the access point. I decided to try out to run a basic throughput test on both the Touchpad and the Laptop. All tests were using the same Comcast consumer broadband connection

DeviceConnectivity TypeLatencyDownload PerformanceUpload Performance
HP Touchpad802.11g Wireless18 milliseconds5.32 Megabits4.78 Megabits
Toshiba dual core Laptop with Ubuntu 10.04 and Firefox 3.6802.11g Wireless13 milliseconds9.46 Megabits4.89 Megabits
Toshiba dual core Laptop with Ubuntu 10.04 and Firefox 3.61 Gigabit ethernet9 milliseconds27.48 Megabits5.09 Megabits

The test runs in flash, and as you can see of course the Touchpad's browser (or flash) is not nearly as fast as the laptop, not too unexpected.

Comparing LAN transfer speeds was even more of a joke of course, I didn't bother involving the Touchpad in this test just the laptop. I used iperf to test throughput(no special options just default settings).

  • Wireless - 7.02 Megabits/second (3.189 milliseconds latency)
  • Wired - 930 Megabits/second (0.3 milliseconds latency)

What honestly surprised me though was over the WAN, how much slower wifi was on the laptop vs wired connection, it's almost 1/3rd the performance on the same laptop/browser. I justed measured to be sure - my laptop's screen (where I believe the antenna is at) is 52 inches from the WRT54G router.

It's "fast enough" for the Touchpad's casual browsing, but certainly wouldn't want to run my home network on it, defeats the purpose of paying for the faster connectivity.

I don't know how typical these results out there. One place I recently worked at was plagued with wireless problems, performance was soo terrible and unreliable. They upgraded the network and I wasn't able to maintain a connection for more than two minutes which sucks for SSH. To make matters worse the vast majority of their LAN was in fact wireless, there was very little cable infrastructure in the office. Smart people hooked up switches and stuff for their own tables which made things more usable, though still a far cry from optimal.

In a world where we are getting even more dense populations and technology continues to penetrate driving more deployments of wifi, I suspect interference problems will only get worse.

I'm sure it's great if the only APs within range are your own, if you live or work at a place that is big enough. But small/medium businesses frequently won't be so lucky, and if you live in a condo or apartment like me, ouch...

My AP is not capable of operating in the 5Ghz range 802.11a/n, that very well could be significantly less congested. I don't know if it is accurate or not but wifi radar claims every AP within range of my laptop(47 at the moment) is 802.11g (same as me). My laptop's specs say it supports 802.11b/g/n, so I'd expect if anyone around me was using N then wifi radar would pick it up, assuming the data being reported by wifi radar is accurate.

Since I am moving in about two weeks I'll wait till I'm at my new apartment before I think more about the possibility of going to a 802.11n capable device for reduced interference. On that note does any of my 3-4 readers have AP suggestions?

Hopefully my new place will get better 4G wireless coverage as well, I already checked the coverage maps and there are two towers within one mile of me, so it all depends on the apartment itself, how much interference is caused by the building and stuff around it.

I'm happy I have stuck with ethernet for as long as I have at my home, and will continue to use ethernet at home and at work wherever possible.

Tagged as: Comments Off

Uptime matters

TechOps Guy: Nate

A friend of mine sent me a link to this xkcd comic and said it reminded him of me, I thought it was fitting given the slogan on the site.

Devotion to Duty


Cleaning the VMCC (3.5) Database

TechOps Guy:

Last weekend encountered a problem with our VMware instance where we could no longer reach a few of our VM's via SSH. No big deal I thought I will just connect to VMCC and jump on the console. So I was mildly annoyed when I couldn't connect, but it was Sunday and there was nothing Product impacted so decide to investigate first thing Monday.

By the time I got in Monday a co-work had already begun investigating why we couldn't connect to VMCC and found the error 'VIM_VCDB' because the 'PRIMARY' filegroup is full. in Event log for the machine hosting VMCC. A little googling gave some pointers to the fix, but most seemed to assume you were a MS SQL DBA and knew what you were doing. Since I am not a MS SQL DBA and did not know what I am doing, I thought I would put what I eventually figured out here in case any other non-DBA's had the same problem. (Note you could completely destroy your VMware installation following these instructions, I would highly recommend you hire an expert to do it)

1.) Download the purge old data SQL script from HERE

2.) If you don't already have it (not being a DBA and all) download SSMSE from HERE

3.) Double- Click the VCDB_table_cleanup_MSSQL script and it will Launch SSMSE

4.) Select the VIM_VCDB database from the drop down

5.) Now click the SSMSE-execute-button button to perform a trail run. You should see something similar to the screen below showing the output of the test run.


From here I will leave as an exercise for the reader to figure out how to enable the cleanup script to actually clean up the database, Hint: read the script comments.


Most Free Credit score sites are a scam

TechOps Guy: Nate

I'm sure both readers of this blog(including me) know this but I just wanted to write about a couple issues on the topic of sites that claim to give you a free credit report(or free credit score). I see these blasted all over TV all the time, well at least on CNBC and CNN where I watch a lot of stuff(no I'm not an investor I just find the news entertaining, long story ask me later).

Anyway my first question revolves around the sites being advertised, just a few minutes ago I saw an ad for the site Throughout the ad they talk about but in the lower left of the ad they show the site What's with the 11 in the name? Why aren't they consistent with the name? Top right of the screen is in fancy letters.

I've seen the same sort of thing with ads for the site they too have added numbers to their domain name in the ads while the actors in the ads never mention the numbers, what's with the numbers?

But the scam comes in to play when you find out(hopefully not before you get the credit report) that you only get that free info if you sign up for a service(in both cases). They do disclose this in the fine print on the ads, and I think on the freetriplescore ad one of the actors even mentions it in a somewhat sly(to me at least) way. But their marketing really drives home the fact that you can get this info for free from them when you cannot.

I think it's likely many people don't notice that actor saying they need to sign up for a service, and probably don't have a DVR so they can pause and read the fine print(assuming the quality is good enough to read, I've seen a lot of fine print on TV that is really hard to read when paused.) And it's these fine print and sly disclosure tricks that make me classify these sites as scams.

I recall a law being passed barring car dealers from using fine print in their TV advertising, I think that should be extended, they should set some sort of standard size of TV and say you can't have text that is smaller than X inches or something.

Last point is there is a place where you can get a free credit report(once per year from each of the major credit reporters), I just looked it up again because they don't advertise as far as I can tell(since they don't make money on it they probably don't have the funds to which is understandable), and I hardly ever hear them mentioned. I think this is the right site it is

You are also of course entitled to receive a copy of a credit report that someone else ran on you say you applied for an apartment or a loan or something, you can write directly to the credit agencies to get a copy of that report. There are probably other times you can get it too, I just remember being told this, and I did it one time about 9 years ago, there was some doctor's office that had something on my credit report that I don't recall having to pay, the doctor was based out of a state I've never been to before, I wrote them asking for more details on why they think I should pay them and a few months later they wrote back saying they removed that item from my report without any explanation, I guess it was a mistake on their end to begin with.

As a Providian..I mean Washington Mutual..I mean now Chase bank customer I did like (note past tense) the ability to check my credit score on their web site for free, never had to request it they just gave it to me and the history over the past 6 months or so. Since Chase acquired them though that feature is gone, oh well. That really was a unique feature among banks that I had relationships with that kept me there.

I'm also a customer with BofA, and the somewhat unique feature I like with them is the ability to generate temporary credit card numbers, I use that feature extensively, whenever possible really. I'm sure lots of banks offer both of these features, but it's not something I was looking for when I signed up for them at the time(many many years ago), and I think both are nice things to have. I should get more for the ~24% interest rates I pay(I don't mind higher interest rates I see it as incentive to pay it off sooner).


Intel doesn’t like wget

TechOps Guy: Nate

I noticed a couple of days ago, while testing out a new proxy system at my company that Intel doesn't like wget. Out of habbit, I usually use wget and sites like or or etc for testing internet connectivity from the command line. It had me running in circles for a little bit trying to troubleshoot the proxy when I realized it was the client that Intel was rejecting. I verified the results on multiple systems on multiple ISPs.

--2009-09-02 11:13:02--
Connecting to||:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-02 11:13:02 ERROR 403: Forbidden.

And from another system, on another ISP(note connecting to a different IP on Intel's side):

--2009-09-02 11:12:27--
Connecting to||:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-02 11:12:28 ERROR 403: Forbidden.

Don't know what Intel has against little ol wget, it's harmless! I tested curl, lynx and of course other GUI browsers and they were all fine. I haven't gone so far as to change my user agent to see if that is related, I don't know how else they might be able to return a 403 though.


It’s not a bug, it’s a feature!

TechOps Guy: Nate

I must be among a tiny minority of people who have automated database snapshots moving between systems on a SAN.

Earlier this year I setup an automated snapshot process to snapshot a production  MySQL database and bring it over to QA. This runs every day, and runs fine as-is. There is another on-demand process to copy byte-for-byte the same production MySQL DB to another QA mysql server(typically run once every month or two, and runs fine too!).

I also setup a job to snapshot all of the production MySQL DBs(3 currently), and bring them to a dedicated "backup" VM which then backs up the data and compresses it onto our NFS cluster. This runs every day, and runs fine as-is.


Apparently they introduced new "intelligence" in vSphere in the storage system that tries to be smarter about what storage devices are present. This totally breaks these automated processes. Because the data on the LUN is different after I remove the LUN, delete the snapshot, create a new one, and re-present the LUN to vSphere it says HEY THERE IS DIFFERENT DATA SO I'LL GIVE IT A UNIQUE UUID (Nevermind the fact that it is the SAME LUN). During that process the guest VM loses connectivity to the original storage(of course) and does not regain connectivity because VSPHERE thinks the LUN is different so doesn't give the VM access to it. The only fix at that point is to power off the VM, delete all of the Raw device maps, re-create all of the raw device maps and then power on the VM again. @#)!#$ No you can't gracefully halt the guest OS because there are missing LUNs and the guest will hang on shutdown.

So I filed a ticket with vmware, the support team worked on it for a couple of weeks, escalating it everywhere, but as far as anyone could tell it's "doing what it's supposed to do". And they can't imagine how this process works in ESX 3.5 except for the fact that ESX 3.5 was more "dumb" when it came to this sort of thing.


With ESX Server 2.5, VMware is encouraging the use of raw device mapping in the following
• When SAN snapshot or other layered applications are run in the virtual machine. Raw
device mapping better enables scalable backup offloading systems using the features
inherent to the SAN.



Anyways there are a few workarounds for these processes going forward:
- Migrate these LUNs to use Software iSCSI instead of Fiber channel, there is a performance hit(not sure how much)
- Keep one/more ESX 3.5 systems around for this type of work
- Use physical servers for things that need automated snapshots

The VMWare support rep sounded about as frustrated with the situation as I was/am. He did appear to try his best, but this behavior by vSphere is just unacceptable.  After all it works flawlessly in ESX 3.5!

WAIT! This broken-ness extends to NFS as well!

I filed another support request on a kinda-sorta-similar issue a couple of weeks ago regarding NFS data stores. Our NFS cluster operates with multiple IP addresses. Many(all?) active-active NFS clusters have at least two IPs (one per controller). In vSphere it once again assigns a unique ID based on the IP address rather than the host name to identify the NFS system. As a result if I use the host name on multiple ESX servers there is a very high likelihood(pretty much guaranteed) that I will not be able to do a migration of a VM that is on NFS from one host to another, because vSphere identifies the volumes differently because they are accessing it via a different IP. And if I try to rename the volume to match what is on the other system it tells me there is already a volume named that(when there is not) so I cannot rename it. The only workaround is to hard code the IP to each host, which is not a good solution because you lose multi-node load balancing at that point. Fortunately I have a Fiber channel SAN as well and have migrated all of my VMs off of NFS onto Fiber Channel, so this particular issue doesn't impact me. But I wanted to illustrate this same sort of behavior with UUIDs is not unique to SAN, it can easily affect NAS as well.

You may not be impacted by the NFS stuff if your NFS system is unable to serve out the same file system over multiple controller systems simultaneously. I believe most fall into this category of being limited to 1 file system per controller at any given point in time. Our NFS cluster does not have this limitation.


It is System Administrator Appreciation Day

TechOps Guy:

The Last Friday in July, so don't forgot to shower your favorite System Administrator with praise and caffeine. Otherwise they might be sleepy when the Gremlins attack your Servers.