Back in September I outline a simple
Perl script to remotely monitor the status of various web servers I manage and report on any failures. One shortcoming of the script is that
it has no memory of the previous state of the websites listed for
polling. Thus, once a site fails, the script will continuously report
on the failure until resolved.
For some, this might be just fine, a
simple repetitive reminder until corrected. For
others however, this might not be ideal. If, for example, the problem
is non-trivial to solve, the last thing one needs is a nagging every
few minutes that the issue has yet to be resolved.
I for one am all for notification
without excessive nagging.
The obvious answer to this dilemma is
to store the previous state of the server such that it can be used to
test against the currently state; if the state of the server has
changed, a notification gets sent. Thus one straightforward notification that something has changed.
As a bonus, by reporting on the change
of state, the script will now report on when the server has come back
online as well as when it has failed. This simple change eliminates
what would have been a manual process previously; notifying
stakeholders that the issue has been resolved.
Since the Perl script is evoked by cron
on a regular basis and terminates once polling is complete, the
“current” state of a site will need to be store in secondary
memory, i.e. on disk, for future comparison. This is pretty
straightforward in Perl:
sub logState ($$) {
my ( $host, $state, $time ) = @_;
# Create a filehandle on our log file
my $fh = FileHandle->new(">> $fileLoc");
if (defined $fh) {
# Print to file the necessary information
# delimited with a colon
print $fh "$host:$state:" .$time->datetime. "\n";
$fh->close;
}
}
With a new Filehandle object the script
opens the file previously assigned to the $fileLoc variable for
appending (the ‘>>’ immediately prior to the variable denotes
write by appending).
If a Filehandle object has been
successfully created, the next step is to write a line to the file
with the information necessary for the next iteration of the monitor
script, specifically the host information and its current state.
Note that each line (\n) in the file
will denote information about a specific site and that the related
information is separated by a colon (:). This will be pertinent later in the code, reading of the log file at the next scheduled
execution of the monitor script:
# Our array of polling sites' previous state
my @hostStates = ();
# Populate said array with information from log file
my $fh = FileHandle->new("< $fileLoc");
while ( <$fh> ) {
my( $line ) = $_;
chomp( $line );
push ( @hostStates, $line );
}
$fh->close;
In this bit of code the goal is to get
the previously logged state of each site and populate an array with
the information. At the moment how each record is delimited isn’t of
concern, but simply that each line is information relating to a
specific site and gets its own node in the array.
Note, since the objective here is to simply read the log file the “<” is used by the filehandle to denote that the file is “read-only” and not “append”.
Once the polling of a specific site
occurs, the first item of concern is determining the site’s previous state. For
that the following bit of code is put to use:
sub getPreviousState ($) {
my ( $host ) = @_;
# For each node in the array do the following
foreach ( @hostStates ) {
my( $line ) = $_;
# Break up the information
# using our delimiter, the colon
my ($domain, $state, $time) = split(/:/, $line, 3);
# If we find our site return the previous state
if ( $domain eq $host ) {
return $state;
}
}
}
In this function each element in the
array is broken down to relevant information using the split
function, which delimits the record by a given character, the colon.
From here it is a simple matter of testing the two states, the
previous and current state before rolling into the notification
process.
The complete, improved remote monitor:
#!/usr/bin/perl
use strict;
use FileHandle;
use Time::Piece;
use LWP::UserAgent;
use Net::Ping;
use Net::Twitter::Lite;
### Start Global Settings ###
my $fileLoc = "/var/log/state.txt";
my @hosts = ( "pdw.weinstein.org", "www.weinstein.org" );
### End Global Settings ###
# Our array of polling sites' previous state
my @hostStates = ();
# Populate said array with information from log file
my $fh = FileHandle->new("< $fileLoc");
while ( <$fh> ) {
my( $line ) = $_;
chomp( $line );
push ( @hostStates, $line );
}
$fh->close;
# Clear out the file by writing anew
my $fh = FileHandle->new("> $fileLoc");
$fh->close;
foreach my $host ( @hosts ) {
my $previousState = getPreviousState( $host );
my $url = "http://$host";
my $ua = LWP::UserAgent->new;
my $response = $ua->get( $url );
my $currentState = $response->code;
my $time = localtime;
# If states are not equal we need to notify someone
if ( $previousState ne $currentState ) {
# Do we have an status code?
if ( $response->code ) {
reportError("$host reports
$response->message.\n");
} else {
# HTTP is not responding,
# is it the network connection down?
my $p = Net::Ping->new("icmp");
if ( $p->ping( $host, 2 )) {
reportError ( "$host is responding,
but Apache is not.\n" );
} else {
reportError ( "$host is unreachable.\n" );
}
}
# Not done yet, we need to log
# the current state for future use
logState( $host, $currentState, $time )
}
sub reportError ($) {
my ( $msg ) = @_;
my $nt = Net::Twitter::Lite->new(
username => $username,
password => $pasword );
my $result = eval { $nt->update( $msg ) };
if ( !$result ) {
# Twitter has failed us,
# need to get the word out still...
smsEmail ( $msg );
}
}
sub smsEmail ($) {
my ( $msg ) = @_;
my $to = "7735551234\@txt.exaple.org";
my $from = "pdw\@weinstein.org";
my $subject = "Service Notification";
my $sendmail = '/usr/lib/sendmail';
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL $msg;
close( MAIL );
}
sub logState ($$) {
my ( $host, $state, $time ) = @_;
# Create a filehandle on our log file
my $fh = FileHandle->new(">> $fileLoc");
if (defined $fh) {
# Print to file the necessary information,
# delimited with a colon
print $fh "$host:$state:" .$time->datetime. "\n";
$fh->close;
}
}
sub getPreviousState ($) {
my ( $host ) = @_;
# For each node in the array do the following
foreach ( @hostStates ) {
my( $line ) = $_;
# Break up the information using our delimiter,
# the colon
my ($domain, $state, $time) = split(/:/, $line, 3);
# If we find our site return the previous state
if ( $domain eq $host ) {
return $state;
}
}
}
