TechOpsGuys.com Availability matters.

Use PowerShell and Task Scheduler to Get Hyper-V CSV Alerts

TechOps Guy: Tycen

I was recently tasked with getting a PowerShell script put together for monitoring CSV (Cluster Shared Volume) sizes on our Hyper-V clusters. We currently have 3 Hyper-V clusters and each cluster has 3 or 4 CSV’s. The problem is that there is seemingly no good way to monitor the volumes since they are not actually driver letter volumes (E:, F:, etc.), but instead are volumes mounted under a folder (e.g., C:\ClusterStorage\Volume1).

So I did a head dive into PowerShell to figure it out. This was my first real attempt at anything PowerShell, so I’m going to file this under the probably-not-the-best-way-to-do-it-but-it-gets-the-job-done category.

I actually created two scripts – one checks weekly and sends a summary report of all the CSV’s, and another one that checks hourly and only sends an email if the free space has dropped below 10%. The frequency (weekly/hourly) is controlled by the scheduled task you create.

I wrote my scripts to run from a central utility server with the scripts and supporting files on a fileshare and to remotely reach out to each of the Hyper-V nodes for the data. I use the DNS name for the cluster (instead of a specific Hyper-V node in the cluster) so that as long as there’s an active node in the cluster, the script will succeed. Rather than hard coding the cluster names into the script, I maintain them in a separate file “cluster_names.txt”.

You need to enable PSRemoting on each Hyper-V node. Cation: when you do that, it will restart WinRM. Also be aware that the VMMAgent service depends on WinRM and will NOT restart when WinRM restarts. So, after you enable PSRemoting, be sure to manually start the VMMAgent service. I used the following command in an elevated privileges PowerShell command window on each Hyper-V node (don’t do this while VM’s are being created or migrated):
PS> Enable-PSRemoting -force ; Start-Service VMMAgent ; sleep 2 ; Get-Service VMMAgent

The scripts are below. I figured it would be best to just paste the full, heavily commented script below instead of walking through the parts step by step.

This first script is the weekly summary check (click the little down triangle/arrow to expand the script):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

#Check CSV free space on Hyper-V clusters and email a summary
#Created by Tycen Stafford - 7/29/2010
#http://www.techopsguys.com
 
#Specify an address to send the alerts to
$toaddress = "whoever@yourdomain.com"
 
#defines a function for sending the message
function sendEmail {
 
$mail = New-Object System.Net.Mail.MailMessage
$mail.From = "utilityhost`@yourdomain.com";
$mail.To.Add("$toaddress");
$mail.Subject = $emailsubject;
$mail.Body = $emailbody
$smtp = New-Object System.Net.Mail.SmtpClient("yoursmtp.yourdomain.com");
$smtp.Send($mail);
}
 
#defines a function for logging into the Hyper-V host and querying it for it's CSV Info.
#The code in this function I found online (don't remember where now) - it pulls back a lot of info and most of it isn't needed
#for this script.
$Function = 
{
	#Hyper-V has some PowerShell modules, we need to import those
	Import-Module FailoverClusters
 
	$objs = @()
 
	#this will get a list of all the CSV's on the Hyper-V node and put them into an array
	$csvs = Get-ClusterSharedVolume
 
	#iterate through the CSV's and get the detailed info
	foreach ( $csv in $csvs ) {
		$csvinfos = $csv | select -Property Name -ExpandProperty SharedVolumeInfo
		foreach ( $csvinfo in $csvinfos ) {
			$obj = New-Object System.Object
			$obj | Add-Member -type NoteProperty -name Name -value $csv.Name
			$obj | Add-Member -type NoteProperty -name Path -value $csvinfo.FriendlyVolumeName
			$obj | Add-Member -type NoteProperty -name Size -value $csvinfo.Partition.Size
			$obj | Add-Member -type NoteProperty -name FreeSpace -value $csvinfo.Partition.FreeSpace
			$obj | Add-Member -type NoteProperty -name UsedSpace -value $csvinfo.Partition.UsedSpace
			$obj | Add-Member -type NoteProperty -name PercentFree -value $csvinfo.Partition.PercentFree
			$objs += $obj
		}
	}
	$objs
}
 
#get a lits of all your clusters - any new clusters should be added to the TXT file referenced in the path
$clusters = Get-Content "\\somefileshare\\Hyper-V_CSV_Scripts\cluster_names.txt"
 
 
#Here's the "meet" of the script
#foreach cluster in the cluster_names.txt file, establish a remote PS Session and get the CSV info and send it in an email
$clusters | % ` {
	$cluster = $_
 
	#for troubleshooting and debugging, I trapped the script in a try command - this could be removed once you get the script polished
	try {
 
		#establish a PSSession for executing remote commands
		$RemoteSession = New-PSSession -ComputerName $cluster
 
		#run a remote command on the server in the PSSession established above
		#the remote command that is ran is the function we defined above that iterates through each CSV for the info
		$CSVInfos = Invoke-Command -Session $RemoteSession -ScriptBlock $Function
 
		#be sure to remove your PSSession
		remove-pssession $remotesession 
 
		#Now we format the email with a body and subject that gives us all the detail we need
		$body = $CSVInfos | ft Name,@{ Label = "Size(GB)" ; Expression = { "{0:N2}" -f ($_.Size/1024/1024/1024) } },@{ Label = "FreeSpace(GB)" ; Expression = { "{0:N2}" -f ($_.FreeSpace/1024/1024/1024) } },@{ Label = "UsedSpace(GB)" ; Expression = { "{0:N2}" -f ($_.UsedSpace/1024/1024/1024) } },@{ Label = "PercentFree" ; Expression = { "{0:N2}" -f ($_.PercentFree) } }
		$emailbody = out-string -inputobject $body
		$emailsubject = "Weekly CSV Summary for Hyper-V Cluster $cluster"
 
		#call the sendemail function defined at the top of the script
		sendEmail 
		}
	catch {
		Write-Host "ERROR OCCURRED"
		Write-Error $_
	}
}

The second script is the check that will email you if the free space has dropped below a certain percent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

#Check CSV free space on Hyper-V clusters and email if the space drops below a certain percentage
#Created by Tycen Stafford - 7/29/2010
#http://www.techopsguys.com
 
#Specify an address to send the alerts to
$toaddress = "whoever@yourdomain.com"
 
#defines a function for sending the message
function sendEmail {
 
$mail = New-Object System.Net.Mail.MailMessage
$mail.From = "utilityhost`@yourdomain.com";
$mail.To.Add("$toaddress");
$mail.Subject = $emailsubject;
$mail.Body = $emailbody
$mail.IsBodyHtml = $true
$smtp = New-Object System.Net.Mail.SmtpClient("yoursmtp.yourdomain.com");
$smtp.Send($mail);
}
 
#get a lits of all our clusters - any new clusters should be added to the TXT file referenced in the path
$clusternames = Get-Content "\\somefileshare\\Hyper-V_CSV_Scripts\cluster_names.txt"
 
#foreach cluster in the cluster_names.txt file, establish a remote PS Session and get the CSVs on that cluster and put them into the array $csvlists
$clusternames | % ` {
	$clustername = $_
	try {
		$RemoteSession1 = New-PSSession -ComputerName $clustername 
		$csvlists = Invoke-Command -Session $RemoteSession1 -ScriptBlock { Import-Module FailoverClusters ; Get-ClusterSharedVolume }
		remove-pssession $RemoteSession1
 
	#define variables: $percentalert is the percentage at which we want an email to be sent; $emailrequency is how often that email should be sent       
	$percentalert = 10
	$emailfrequency = -48
 
	#for each CSV gathered above, re-establish a remote PS Session (expensive and doesn't scale, I know) and get the percent free of that CSV.
	foreach ($csvlist in $csvlists) {
			$RemoteSession2 = New-PSSession -ComputerName $clustername 
			$csvpercentfree = Invoke-Command -Session $RemoteSession2 -ScriptBlock { param([string]$csvlist) Import-Module FailoverClusters ; Get-ClusterSharedVolume  $csvlist | select -Expand SharedVolumeInfo | select -Expand Partition | select -Expand PercentFree } -ArgumentList $csvlist
			remove-pssession $RemoteSession2
 
			#check the percent free of the CSV with your threshold defined in $percentalert
			if ($csvpercentfree -lt $percentalert) {
 
				#format an email ($emailbody and $emailsubject).
				$emailbody = "<FONT FACE=`"Arial`">One of the CSVs in <b>"+$clustername+"</b> has dropped below "+$percentalert+" Percent!<br><br>CSV: <b>"+$csvlist+"</b><br>Percent Free: <b>"+$csvpercentfree+"%</b><br></b></font>"
				$emailsubject = "Hyper-V Cluster "+$clustername+" CSV Free Space Alert"
 
				#the email control file keeps you from getting alerts too often - the script checks the timestamp of the file to know when the last email was sent
				#you'll need to manually create each email controll file - I'm sure there's a way for PS to do it, but I didn't spend the time to try it
				$emailcontrolfile = "\\somefileshare\\Hyper-V_CSV_Scripts\emailcontrolfiles\"+$csvlist+".txt"
				$lastemailed = (Get-Item $emailcontrolfile).LastWriteTime
 
				#if the timestamp of the control file for that CSV is older than X hours ($emailfrequency), send an email and update the time stamp of the control file
				if ($lastemailed -lt ($(Get-Date).addhours($emailfrequency))) {
					sendEmail
					$(Get-Item $emailcontrolfile).lastwritetime=$(get-date)
				}
			}
		}
	}
	catch {
		Write-Host "ERROR OCCURRED"
		Write-Error $_
	}
}

8 Things Startups Must Know Before Signing a Lease

TechOps Guy: Jason

I’ve had the opportunity to perform office selection and relocation a number of times in my career and I wanted to give the tech community some quick and dirty lessons when considering new office space.

#1 – Direct vs. Sublease
Before committing your company to 5+ year lease you might want to consider subleasing from an existing company that is downsizing. In this market cash conscious startups can easily negotiate short and long-term subleases. Remember shorter is better and allows for greater flexibility and potentially an easier exit should you become acquired.

#2 – Hire a Tenant Broker, Not a Commercial Real Estate Agent
I know that everyone has a friend that is in the business, but remember you need an advocate. Tenant Brokers do not offer their services to Building Owners so they are free to aggressively negotiate on your behalf.

#3 – Whenever Possible Create Pocket Space
Creating a pocket space allows you to grow into your new office without paying for the full office footprint up front. Can you set aside 500/1000/2000 sq ft for future growth? If so, add it to the lease.

#4 – Controlling Costs During Build Out
Assign a high performing individual to manage vendors and change requests. Make this person responsible for managing the day-to-day decisions that arise as you complete your build out. Think about it, every little change request can add up to a bunch of dollars quickly.

#5 – Do Not Create Specialized Offices or Spaces for Individual Contributors
The fact is your workforce changes over time, your office space generally does not unless you are willing to cough up money for tenant improvements as your needs change. Keep your office flexible and the ability to scale as you add more employees.

#6 – Termination Clauses
Planning on being acquired before your lease runs out? Are you afraid you will grow out of the space the building provided? See if you can be given the first right of refusal on adjacent space, or spaces within the building. If they cannot accommodate your growth then add that to the termination clause.

#7 – Commuting & Service Friendly
Is this location commuter friendly? Can your employees easily access basic services including great coffee shops and lunch places? Does your space have a kitchen and fridge? Your talent does care where you are located and basic amenities.

#8 – Employee Safety
As the employer you need to think carefully about a building’s location and parking facilities and their safety. Not everyone is the same, but consider what it might be like for one of your employees to be leaving the office at 9pm – will your employees feel safe?

Uptime matters

TechOps Guy: Nate

A friend of mine sent me a link to this xkcd comic and said it reminded him of me, I thought it was fitting given the slogan on the site.

Going GREEN at the Datacenter? Better check your server temperatures!

TechOps Guy: Jason

It seems innocuous enough, but we recently stumbled across the following Windows Event Log error indicating our servers were running at a very cool 8 degrees C…in fact, too cool!

So you’re asking what’s the GREEN connection? It turns out when we contacted our datacenter provider, Internap/Sabey they replied with the following:

“In an effort to be “green” we have been in economizer mode drawing cooling from the chilly outside air. We have adjusted our set points to correct this. I’ve left a message with the customer as well.

Thanks and let me know if there are any other questions”

In short, definitely make sure you have temperature monitoring, not just for heat, but for cold.

Cleaning the VMCC (3.5) Database

TechOps Guy: Dave

Last weekend encountered a problem with our VMware instance where we could no longer reach a few of our VM’s via SSH. No big deal I thought I will just connect to VMCC and jump on the console. So I was mildly annoyed when I couldn’t connect, but it was Sunday and there was nothing Product impacted so decide to investigate first thing Monday.

By the time I got in Monday a co-work had already begun investigating why we couldn’t connect to VMCC and found the error ‘VIM_VCDB’ because the ‘PRIMARY’ filegroup is full. in Event log for the machine hosting VMCC. A little googling gave some pointers to the fix, but most seemed to assume you were a MS SQL DBA and knew what you were doing. Since I am not a MS SQL DBA and did not know what I am doing, I thought I would put what I eventually figured out here in case any other non-DBA’s had the same problem. (Note you could completely destroy your VMware installation following these instructions, I would highly recommend you hire an expert to do it)

1.) Download the purge old data SQL script from HERE

2.) If you don’t already have it (not being a DBA and all) download SSMSE from HERE

3.) Double- Click the VCDB_table_cleanup_MSSQL script and it will Launch SSMSE

4.) Select the VIM_VCDB database from the drop down

5.) Now click the button to perform a trail run. You should see something similar to the screen below showing the output of the test run.

From here I will leave as an exercise for the reader to figure out how to enable the cleanup script to actually clean up the database, Hint: read the script comments.

Most Free Credit score sites are a scam

TechOps Guy: Nate

I’m sure both readers of this blog(including me) know this but I just wanted to write about a couple issues on the topic of sites that claim to give you a free credit report(or free credit score). I see these blasted all over TV all the time, well at least on CNBC and CNN where I watch a lot of stuff(no I’m not an investor I just find the news entertaining, long story ask me later).

Anyway my first question revolves around the sites being advertised, just a few minutes ago I saw an ad for the site freescore.com. Throughout the ad they talk about freescore.com but in the lower left of the ad they show the site freescore11.com. What’s with the 11 in the name? Why aren’t they consistent with the name? Top right of the screen is freescore.com in fancy letters.

I’ve seen the same sort of thing with ads for the site freetriplescore.com they too have added numbers to their domain name in the ads while the actors in the ads never mention the numbers, what’s with the numbers?

But the scam comes in to play when you find out(hopefully not before you get the credit report) that you only get that free info if you sign up for a service(in both cases). They do disclose this in the fine print on the ads, and I think on the freetriplescore ad one of the actors even mentions it in a somewhat sly(to me at least) way. But their marketing really drives home the fact that you can get this info for free from them when you cannot.

I think it’s likely many people don’t notice that actor saying they need to sign up for a service, and probably don’t have a DVR so they can pause and read the fine print(assuming the quality is good enough to read, I’ve seen a lot of fine print on TV that is really hard to read when paused.) And it’s these fine print and sly disclosure tricks that make me classify these sites as scams.

I recall a law being passed barring car dealers from using fine print in their TV advertising, I think that should be extended, they should set some sort of standard size of TV and say you can’t have text that is smaller than X inches or something.

Last point is there is a place where you can get a free credit report(once per year from each of the major credit reporters), I just looked it up again because they don’t advertise as far as I can tell(since they don’t make money on it they probably don’t have the funds to which is understandable), and I hardly ever hear them mentioned. I think this is the right site it is annualcreditreport.com.

You are also of course entitled to receive a copy of a credit report that someone else ran on you say you applied for an apartment or a loan or something, you can write directly to the credit agencies to get a copy of that report. There are probably other times you can get it too, I just remember being told this, and I did it one time about 9 years ago, there was some doctor’s office that had something on my credit report that I don’t recall having to pay, the doctor was based out of a state I’ve never been to before, I wrote them asking for more details on why they think I should pay them and a few months later they wrote back saying they removed that item from my report without any explanation, I guess it was a mistake on their end to begin with.

As a Providian..I mean Washington Mutual..I mean now Chase bank customer I did like (note past tense) the ability to check my credit score on their web site for free, never had to request it they just gave it to me and the history over the past 6 months or so. Since Chase acquired them though that feature is gone, oh well. That really was a unique feature among banks that I had relationships with that kept me there.

I’m also a customer with BofA, and the somewhat unique feature I like with them is the ability to generate temporary credit card numbers, I use that feature extensively, whenever possible really. I’m sure lots of banks offer both of these features, but it’s not something I was looking for when I signed up for them at the time(many many years ago), and I think both are nice things to have. I should get more for the ~24% interest rates I pay(I don’t mind higher interest rates I see it as incentive to pay it off sooner).

Intel doesn’t like wget

TechOps Guy: Nate

I noticed a couple of days ago, while testing out a new proxy system at my company that Intel doesn’t like wget. Out of habbit, I usually use wget and sites like intel.com or cnn.com or netscape.com etc for testing internet connectivity from the command line. It had me running in circles for a little bit trying to troubleshoot the proxy when I realized it was the client that Intel was rejecting. I verified the results on multiple systems on multiple ISPs.

–2009-09-02 11:13:02–  http://www.intel.com/
Resolving www.intel.com… 208.50.77.158, 208.50.77.167
Connecting to www.intel.com|208.50.77.158|:80… connected.
HTTP request sent, awaiting response… 403 Forbidden
2009-09-02 11:13:02 ERROR 403: Forbidden.

And from another system, on another ISP(note connecting to a different IP on Intel’s side):

–2009-09-02 11:12:27–  http://www.intel.com/
Resolving www.intel.com… 96.17.8.8, 96.17.8.80
Connecting to www.intel.com|96.17.8.8|:80… connected.
HTTP request sent, awaiting response… 403 Forbidden
2009-09-02 11:12:28 ERROR 403: Forbidden.

Don’t know what Intel has against little ol wget, it’s harmless! I tested curl, lynx and of course other GUI browsers and they were all fine. I haven’t gone so far as to change my user agent to see if that is related, I don’t know how else they might be able to return a 403 though.

It is System Administrator Appreciation Day

TechOps Guy: Dave

The Last Friday in July, so don’t forgot to shower your favorite System Administrator with praise and caffeine. Otherwise they might be sleepy when the Gremlins attack your Servers.

http://www.sysadminday.com/index2009.html