TechOpsGuys.com Diggin' technology every day

23Feb/100

Uptime matters

TechOps Guy: Nate

A friend of mine sent me a link to this xkcd comic and said it reminded him of me, I thought it was fitting given the slogan on the site.

Devotion to Duty

Filed under: Uncategorized No Comments
7Oct/091

Going GREEN at the Datacenter? Better check your server temperatures!

TechOps Guy: Jason

It seems innocuous enough, but we recently stumbled across the following Windows Event Log error indicating our servers were running at a very cool 8 degrees C...in fact, too cool!

Your Server is 8 degrees C.

So you're asking what's the GREEN connection? It turns out when we contacted our datacenter provider, Internap/Sabey they replied with the following:

"In an effort to be "green" we have been in economizer mode drawing cooling from the chilly outside air. We have adjusted our set points to correct this. I've left a message with the customer as well.

Thanks and let me know if there are any other questions"

In short, definitely make sure you have temperature monitoring, not just for heat, but for cold.

Filed under: Uncategorized 1 Comment
2Oct/090

Cleaning the VMCC (3.5) Database

TechOps Guy:

Last weekend encountered a problem with our VMware instance where we could no longer reach a few of our VM's via SSH. No big deal I thought I will just connect to VMCC and jump on the console. So I was mildly annoyed when I couldn't connect, but it was Sunday and there was nothing Product impacted so decide to investigate first thing Monday.

By the time I got in Monday a co-work had already begun investigating why we couldn't connect to VMCC and found the error 'VIM_VCDB' because the 'PRIMARY' filegroup is full. in Event log for the machine hosting VMCC. A little googling gave some pointers to the fix, but most seemed to assume you were a MS SQL DBA and knew what you were doing. Since I am not a MS SQL DBA and did not know what I am doing, I thought I would put what I eventually figured out here in case any other non-DBA's had the same problem. (Note you could completely destroy your VMware installation following these instructions, I would highly recommend you hire an expert to do it)

1.) Download the purge old data SQL script from HERE

2.) If you don't already have it (not being a DBA and all) download SSMSE from HERE

3.) Double- Click the VCDB_table_cleanup_MSSQL script and it will Launch SSMSE
sqlserver-connect

4.) Select the VIM_VCDB database from the drop down
SSMSE-screenshot

5.) Now click the SSMSE-execute-button button to perform a trail run. You should see something similar to the screen below showing the output of the test run.

cleanup-complete

From here I will leave as an exercise for the reader to figure out how to enable the cleanup script to actually clean up the database, Hint: read the script comments.

Filed under: Uncategorized No Comments
10Sep/099

Most Free Credit score sites are a scam

TechOps Guy: Nate

I'm sure both readers of this blog(including me) know this but I just wanted to write about a couple issues on the topic of sites that claim to give you a free credit report(or free credit score). I see these blasted all over TV all the time, well at least on CNBC and CNN where I watch a lot of stuff(no I'm not an investor I just find the news entertaining, long story ask me later).

Anyway my first question revolves around the sites being advertised, just a few minutes ago I saw an ad for the site freescore.com. Throughout the ad they talk about freescore.com but in the lower left of the ad they show the site freescore11.com. What's with the 11 in the name? Why aren't they consistent with the name? Top right of the screen is freescore.com in fancy letters.

I've seen the same sort of thing with ads for the site freetriplescore.com they too have added numbers to their domain name in the ads while the actors in the ads never mention the numbers, what's with the numbers?

But the scam comes in to play when you find out(hopefully not before you get the credit report) that you only get that free info if you sign up for a service(in both cases). They do disclose this in the fine print on the ads, and I think on the freetriplescore ad one of the actors even mentions it in a somewhat sly(to me at least) way. But their marketing really drives home the fact that you can get this info for free from them when you cannot.

I think it's likely many people don't notice that actor saying they need to sign up for a service, and probably don't have a DVR so they can pause and read the fine print(assuming the quality is good enough to read, I've seen a lot of fine print on TV that is really hard to read when paused.) And it's these fine print and sly disclosure tricks that make me classify these sites as scams.

I recall a law being passed barring car dealers from using fine print in their TV advertising, I think that should be extended, they should set some sort of standard size of TV and say you can't have text that is smaller than X inches or something.

Last point is there is a place where you can get a free credit report(once per year from each of the major credit reporters), I just looked it up again because they don't advertise as far as I can tell(since they don't make money on it they probably don't have the funds to which is understandable), and I hardly ever hear them mentioned. I think this is the right site it is annualcreditreport.com.

You are also of course entitled to receive a copy of a credit report that someone else ran on you say you applied for an apartment or a loan or something, you can write directly to the credit agencies to get a copy of that report. There are probably other times you can get it too, I just remember being told this, and I did it one time about 9 years ago, there was some doctor's office that had something on my credit report that I don't recall having to pay, the doctor was based out of a state I've never been to before, I wrote them asking for more details on why they think I should pay them and a few months later they wrote back saying they removed that item from my report without any explanation, I guess it was a mistake on their end to begin with.

As a Providian..I mean Washington Mutual..I mean now Chase bank customer I did like (note past tense) the ability to check my credit score on their web site for free, never had to request it they just gave it to me and the history over the past 6 months or so. Since Chase acquired them though that feature is gone, oh well. That really was a unique feature among banks that I had relationships with that kept me there.

I'm also a customer with BofA, and the somewhat unique feature I like with them is the ability to generate temporary credit card numbers, I use that feature extensively, whenever possible really. I'm sure lots of banks offer both of these features, but it's not something I was looking for when I signed up for them at the time(many many years ago), and I think both are nice things to have. I should get more for the ~24% interest rates I pay(I don't mind higher interest rates I see it as incentive to pay it off sooner).

Filed under: Uncategorized 9 Comments
2Sep/090

Intel doesn’t like wget

TechOps Guy: Nate

I noticed a couple of days ago, while testing out a new proxy system at my company that Intel doesn't like wget. Out of habbit, I usually use wget and sites like intel.com or cnn.com or netscape.com etc for testing internet connectivity from the command line. It had me running in circles for a little bit trying to troubleshoot the proxy when I realized it was the client that Intel was rejecting. I verified the results on multiple systems on multiple ISPs.

--2009-09-02 11:13:02--  http://www.intel.com/
Resolving www.intel.com... 208.50.77.158, 208.50.77.167
Connecting to www.intel.com|208.50.77.158|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-02 11:13:02 ERROR 403: Forbidden.

And from another system, on another ISP(note connecting to a different IP on Intel's side):

--2009-09-02 11:12:27--  http://www.intel.com/
Resolving www.intel.com... 96.17.8.8, 96.17.8.80
Connecting to www.intel.com|96.17.8.8|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2009-09-02 11:12:28 ERROR 403: Forbidden.

Don't know what Intel has against little ol wget, it's harmless! I tested curl, lynx and of course other GUI browsers and they were all fine. I haven't gone so far as to change my user agent to see if that is related, I don't know how else they might be able to return a 403 though.

Filed under: Uncategorized No Comments
18Aug/090

It’s not a bug, it’s a feature!

TechOps Guy: Nate

I must be among a tiny minority of people who have automated database snapshots moving between systems on a SAN.

Earlier this year I setup an automated snapshot process to snapshot a production  MySQL database and bring it over to QA. This runs every day, and runs fine as-is. There is another on-demand process to copy byte-for-byte the same production MySQL DB to another QA mysql server(typically run once every month or two, and runs fine too!).

I also setup a job to snapshot all of the production MySQL DBs(3 currently), and bring them to a dedicated "backup" VM which then backs up the data and compresses it onto our NFS cluster. This runs every day, and runs fine as-is.

ENTER VMWARE VSPHERE.

Apparently they introduced new "intelligence" in vSphere in the storage system that tries to be smarter about what storage devices are present. This totally breaks these automated processes. Because the data on the LUN is different after I remove the LUN, delete the snapshot, create a new one, and re-present the LUN to vSphere it says HEY THERE IS DIFFERENT DATA SO I'LL GIVE IT A UNIQUE UUID (Nevermind the fact that it is the SAME LUN). During that process the guest VM loses connectivity to the original storage(of course) and does not regain connectivity because VSPHERE thinks the LUN is different so doesn't give the VM access to it. The only fix at that point is to power off the VM, delete all of the Raw device maps, re-create all of the raw device maps and then power on the VM again. @#)!#$ No you can't gracefully halt the guest OS because there are missing LUNs and the guest will hang on shutdown.

So I filed a ticket with vmware, the support team worked on it for a couple of weeks, escalating it everywhere, but as far as anyone could tell it's "doing what it's supposed to do". And they can't imagine how this process works in ESX 3.5 except for the fact that ESX 3.5 was more "dumb" when it came to this sort of thing.

ITS RAW FOR A REASON, DON'T TRY TO BE SMART WHEN IT COMES TO A RAW DEVICE MAP, THAT'S WHY IT'S RAW.

http://www.vmware.com/pdf/esx25_rawdevicemapping.pdf

With ESX Server 2.5, VMware is encouraging the use of raw device mapping in the following
situations:
• When SAN snapshot or other layered applications are run in the virtual machine. Raw
device mapping better enables scalable backup offloading systems using the features
inherent to the SAN.

[..]

HELLO ! SAN USER HERE TRYING TO OFFLOAD BACKUPS!

Anyways there are a few workarounds for these processes going forward:
- Migrate these LUNs to use Software iSCSI instead of Fiber channel, there is a performance hit(not sure how much)
- Keep one/more ESX 3.5 systems around for this type of work
- Use physical servers for things that need automated snapshots

The VMWare support rep sounded about as frustrated with the situation as I was/am. He did appear to try his best, but this behavior by vSphere is just unacceptable.  After all it works flawlessly in ESX 3.5!

WAIT! This broken-ness extends to NFS as well!

I filed another support request on a kinda-sorta-similar issue a couple of weeks ago regarding NFS data stores. Our NFS cluster operates with multiple IP addresses. Many(all?) active-active NFS clusters have at least two IPs (one per controller). In vSphere it once again assigns a unique ID based on the IP address rather than the host name to identify the NFS system. As a result if I use the host name on multiple ESX servers there is a very high likelihood(pretty much guaranteed) that I will not be able to do a migration of a VM that is on NFS from one host to another, because vSphere identifies the volumes differently because they are accessing it via a different IP. And if I try to rename the volume to match what is on the other system it tells me there is already a volume named that(when there is not) so I cannot rename it. The only workaround is to hard code the IP to each host, which is not a good solution because you lose multi-node load balancing at that point. Fortunately I have a Fiber channel SAN as well and have migrated all of my VMs off of NFS onto Fiber Channel, so this particular issue doesn't impact me. But I wanted to illustrate this same sort of behavior with UUIDs is not unique to SAN, it can easily affect NAS as well.

You may not be impacted by the NFS stuff if your NFS system is unable to serve out the same file system over multiple controller systems simultaneously. I believe most fall into this category of being limited to 1 file system per controller at any given point in time. Our NFS cluster does not have this limitation.

Tagged as: , , No Comments
31Jul/090

It is System Administrator Appreciation Day

TechOps Guy:

The Last Friday in July, so don't forgot to shower your favorite System Administrator with praise and caffeine. Otherwise they might be sleepy when the Gremlins attack your Servers.

http://www.sysadminday.com/index2009.html

Filed under: Uncategorized No Comments