TechOpsGuys.com Availability matters.

Acknowledge Nagios Alerts Via Email Replies

TechOps Guy: Tycen

Monitoring should be annoying by design – when something is broken, we need to fix it and we need to be reminded it needs fixing until it gets fixed. That’s why we monitor in the first place. In that vein, I’ve configured our Nagios server to notify every hour for most alerts. However, there are times when a certain alert can be ignored for a while and I might not have a computer nearby to acknowledge it.

The solution: acknowledge Nagios alerts via email. A quick reply on my smartphone and I’m done.

Setting it up is fairly simple and involves a few components: an MTA (Postfix in my case), procmail (might need to install it), a perl script and the nagios.cmd file. I used the info in this post to get me started. My instructions below were done on two different CentOS 5.4 installs running Nagios 3.0.6 and Nagios 3.2.0.

Procmail
Make a /home/nagios/.procmailrc file (either su to the nagios user or chown to nagios:nagios afterwards) and paste in the following:

LOGFILE=$HOME/.procmailrc.log
MAILDIR=$HOME/Mail
VERBOSE=yes
PATH=/usr/bin
:0
* ^Subject:[    ]*\/[^  ].*
| /usr/lib/nagios/eventhandlers/processmail "${MATCH}"

Postfix
Tell Postfix to use procmail by adding the the following line to /etc/postfix/main.cf (restart Postfix when finished):
mailbox_command = /usr/bin/procmail
You might want to search your main.cf file for mailbox_command to make sure procmail isn’t already configured/turned on. You also might want to do whereis procmail to make sure it’s in the /usr/bin folder. If your Nagios server hasn’t previously been configured to receive email, you’ve got some configuration to do – that’s outside of the scope of this article, but I would suggest getting that up and running first.

Perl Script
Next up is the perl script that procmail references. Create a /usr/lib/nagios/eventhandlers/processmail file and chmod 755 it – paste in the code below:

#!/usr/bin/perl
 
$correctpassword = 'whatever';   # more of a sanity check than a password and can be anything
$subject = "$ARGV[0]";
$now = `/bin/date +%s`;
chomp $now;
$commandfile = '/usr/local/nagios/var/rw/nagios.cmd';
 
if ($subject =~ /Host/ ){      # this parses the subject of your email
        ($password, $what, $junk, $junk, $junk, $junk, $junk, $host) = split(/ /, $subject);
        ($host) = ($host) =~ /(.*)\!/;
} else {
        ($foo, $bar) = split(/\//, $subject);
        ($password, $what, $junk, $junk, $junk, $junk, $junk, $host) = split(/\ /, $foo);
        ($service) = $bar =~ /^(.*) is.*$/;
}
 
$password =~ s/^\s+//;
$password =~ s/\s+$//;
 
print "$password\t$what\t$host\t$service\n";
 
unless ($password =~ /$correctpassword/i) {
        print "exiting...wrong password\n";
        exit 1;
}
 
# ack - this is where the acknowledgement happens
# you could get creative with this and pass all kinds of things via email
# a list of external commands here: http://www.nagios.org/development/apis/externalcommands/
if ($subject =~ /Host/ ) {
        $ack =
"ACKNOWLEDGE_HOST_PROBLEM;$host;1;1;1;email;email;acknowledged through email";
} else {
        $ack = "ACKNOWLEDGE_SVC_PROBLEM;$host;$service;1;1;1;email;acknowledged through email";
}
 
if ($what =~ /ack/i) {
        sub_print("$ack");
} else {
        print "no valid commands...exiting\n";
        exit 1;
}
 
 
sub sub_print {
        $narf=shift;
        open(F, ">$commandfile") or die "cant";
        print F "[$now] $narf\n";
        close F;
}

The script above assumes certain things about how your email subject line is formatted and you might have to tweak it if you’ve done much/any customization to the Notification commands in the default commands.cfg file. One thing you will need to change is the “Host” variable. The default is to put Host: $HOSTALIAS$ in the subject – you’ll need to replace that with $HOSTNAME$ as that is what the nagios.cmd file expects. If you don’t change that, the perl script above will pass the $HOSTALIAS$ to the nagios.cmd file and it won’t know what to do with it. Below is a sample of my notify-service-by-email command:

define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nComment: $SERVICEACKCOMMENT$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nMonitoring Page: http://nagios1/nagios\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTNAME$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

Example
So, when i get an alert that has a subject something like this:
** PROBLEM alert - server1/CPU Load is WARNING **
I can just reply and add “whatever ack” to the beginning of the subject line:
whatever ack RE: ** PROBLEM alert - server1/CPU Load is WARNING **
and the alert will be acknowledged.

Troubleshooting
As I said earlier, you will want to make sure Postfix is configured correctly for receiving email for the Nagios user – this might be an area where you’ll have issues if it’s not set up correctly. The other thing that fouled me up a few times was the Notification command section I mentioned above. By passing commands directly to the nagios.cmd file and by watching the log files, you should be able to spot any misconfigs.

Uptime of various web properties

TechOps Guy: Nate

Came across a post on Techcrunch, which then lead me to Alertsite, which seems to maintain a list of various web sites in various industries and their average uptime and response time. I thought it was interesting at least that Amazon is up only 97% of the time, and LinkedIn up only 95% of the time for examples. Kind of puts things in perspective I think, an increasing number of people and organizations are “demanding” higher levels of uptime, while it’s certainly achievable it seems in many cases the costs are just not worth it.  Taking it to an extreme level, this topic reminds me of this article written several years ago by our best friends at The Register.

When Microsoft goofed the DNS settings on its microsoft.com servers recently, he figured the site would have to be up for the next two hundred years to achieve five-nines uptime.

Don’t know why I remember things like that but can’t remember other things like birthdays.

1 Billion events in Splunk

TechOps Guy: Nate

I was on a conference call with Splunk about a month or so ago, we recently bought it after using it off and on for a while. One thing that stuck out to me on that call was the engineer’s excitement around being able to show off a system that had a billion events in it. I started a fresh Splunk database in early June 2009 I think it was, and recently we passed 1 billion events. The index/DB(whatever you want to call it) just got to about 100GB(the below screenshot is a week or two old). The system is still pretty quick too. Running on a simple dual Xeon system with 8GB memory, and a software iSCSI connection to the SAN.

We have something like 400 hosts logging to it(just retired about 100 additional ones about a month ago, going to retire another 80-100 in the coming weeks as we upgrade hardware). It’s still not fully deployed right now about 99% of the data is from syslog.

Upgraded to Splunk v4 the day it came out, it has some nice improvements, filed a bug the day it came out too(well a few), but the most annoying one is I can’t login to v4 with Mozilla browsers(nobody in my company can). Only with IE. We suspect it’s some behavioral issue with our really basic Apache reverse proxy and Splunk. The support guys are looking at it still. That and both their Cisco and F5 apps do not show any data despite having millions of log events from both Cisco and F5 devices in our index. They are looking into that too.

1 billion logged events

Are My Emails Getting Through? The Need to Monitor Email Deliverability

TechOps Guy: Jason

How do we define whether an email was delivered? While this is a simple question with what would seem like a simple solution is actually very difficult to accomplish, especially if you wish to monitor email deliverability across many of the top email providers. To our discussion, we will limit the definition of email deliverability to creating a message with plain-text or HTML content and have it delivered to the recipient’s Inbox, not the Junk or Spam folders. The simplest, although time intensive method for which to test email deliverability is to directly send a message from the Sender to a test Recipient at a particular email provider and log in as the test Recipient to see if the message was delivered as expected in its true form. Obviously, this does not scale.

What is the ideal solution?

We live and work in an environment that requires consistent process improvement–this manual verification of email delivery is very time intensive and screams for automation, but automation of what?

Define the business problem

A company needs to send out important emails to subscribers, marketing emails, etc.; subsequently they need to be sure that their messages were received in tact, and delivered to the Inbox bypassing Spam and Junk mail filters.

Summary of the Solution

• Email Providers
• Application to Monitor Deliverability
• Controls to Determine the “Health” of Email Deliverability

Email Providers
What are the top email providers for your subscribers’ based email? For discussion purposes we will use the following which is a list of the most trafficked mail servers from Alexa.com on 1/7/2007.

• Yahoo
• Google
• Hotmail
• Comcast
• Earthlink
• Excite

* Please pay special attention to your target audience as they will likely indicate the email services for which you will need to verify deliverability.

Application to Monitor Deliverability
This is the meat of the discussion. You need to build an application that can perform the following series of tests and report back a Boolean RED = Bad; GREEN = Good

• Application must run as a service on the web server
• Ability to send an email mimicking the same format as your subscriber/marketing emails
• For a list of defined Email Providers the application needs to send an email to each Recipient from the Sender *
• The application must then log into each Email Provider as the Recipient and verify that the message can be read in the Inbox, not Spam or Junk mail folders
• The application must then report back to the monitoring system whether the email was delivered as expected; RED/GREEN
• The monitoring system can then aggregate all of the responses across the Email Providers and report on the deliverability “Health”
• Finally, when the monitoring system notices a RED or Bad response it can alert via email, pager and/or other notification to the Technical Operations team to address the issue

* Note the sender needs to be the same sender as your outgoing subscriber/marketing emails.

For an example of the above requirements, please refer to the diagram below.

Controls to Determine the “Health” of Email Deliverability

The application described above can be controlled directly via EmailProviders.xml file format. Once the application is running as a service on the web server, it is a matter of determining what types of controls you wish to take advantage of to determine and monitor the “Health” of your email deliverability. The most obvious controls are:

• Email Providers you wish to check

• Username/Passwords for each test recipient

• Interval you wish to test the email deliverability

• Notification method you wish to use in the event of an error

I hope you found this useful as a discussion point in how to monitor your company’s direct email “Health”.

Thanks,

Jason

The Difficulty of Intelligent Application Monitoring

TechOps Guy: Jason

It’s become increasingly more important for companies with online applications to have detailed monitoring. No longer are the days when we can monitor drive space, services and ICMP responses to verify availability. One of the biggest assets a company with an online application can have is the ability to understand how the application is behaving at any given time under any conditions. Traditionally this breaks down into 4 areas:

1.) Network

2.) Server

3.) Application

4.) Integration

Most IT professionals are very familiar with the first two categories; as these monitoring capabilities are available in almost every software solution off of the shelf. The old school of thought would be to setup ICMP monitors for the IPs of the application and make sure that you received a prompt reply. The second part of the old school strategy would be to monitor critical items on the server such as disk space, % processor time, status of services running, etc. While both of these strategies are still very much used today, what separates a more robust monitoring solution is adding application and integration level intelligence.

Building in Application/Integration Level Monitoring:

First, let’s consider the following very basic web server configuration. You have an application named WebApplication1 running on a web server named WebServer1. WebApplication1 is a simple user community that allows users to register (Register.jsp), login (Login.jsp) and review postings by other users (ViewPage.jsp) and create/update/delete postings of their own (PageFunctions.jsp). The registration, login and password (Password.jsp) pages are protected by SSL encryption (port 443); the rest of the user community utilizes HTTP (port 80). In addition, WebServer1 runs on another application called AppMonitor1 on port 7001 to report the health of the WebApplication1 application via Monitor.jsp.

Sample Data Flow is given below:

WebServer1’s Web Configuration:

WebAppliation1 is running on WebServer1/2, but could theoretically be scaled out to XX number of servers. In this example, WebApplication1 is running version r1.0 under both ports 80 and 443 while AppMonitor1 is running version r1.1 on port 7001.

For example let’s take the following application under consideration:

Internet –> Router –> Firewall (Not pictured) –> NAT’d addresses –> Load Balancer –> Web Server(s) –> Application Server(s) –> Database Server(s) –> Fail-over
Systems (Not all pictured)

The above architecture reflects one example of how application level monitoring could be implemented. In this example we have a shell script or compiled exe running as a service known as AppMonitor1.sh running on Monitor1 that would poll the WebServer1/2/XX pool to check the health of the application at given intervals. If for any reason the AppMonitor1 script cannot return a response then the script results in an email alert being sent out to the Technical Operations team. Under normal circumstances the AppMonitor1 service will be able to return
results showing the status/behavior of critical application indicators as shown in the example below. This service is then run at regular intervals.

Ideally you need to understand how specifically your application is behaving and not just a web page with a returned dataset that says “OK/GREEN”.