{"id":195,"date":"2009-11-12T16:00:02","date_gmt":"2009-11-13T00:00:02","guid":{"rendered":"http:\/\/35.225.155.113\/blog\/index.php\/2009\/11\/12\/a_stepped_up_remote_apache_monitor_in_perl\/"},"modified":"2019-10-13T12:56:15","modified_gmt":"2019-10-13T19:56:15","slug":"a-stepped-up-remote-apache-monitor-in-perl","status":"publish","type":"post","link":"https:\/\/www.weinstein.org\/blog\/index.php\/2009\/11\/a-stepped-up-remote-apache-monitor-in-perl.html","title":{"rendered":"A Stepped Up Remote Apache Monitor in Perl"},"content":{"rendered":"

Back in September I outline a simple
\nPerl script to remotely monitor the status of various web servers<\/a> I manage and report on any failures. One shortcoming of the script is that
\nit has no memory of the previous state of the websites listed for
\npolling. Thus, once a site fails, the script will continuously report
\non the failure until resolved.<\/p>\n

For some, this might be just fine, a
\nsimple repetitive reminder until corrected. For
\nothers however, this might not be ideal. If, for example, the problem
\nis non-trivial to solve, the last thing one needs is a nagging every
\nfew minutes that the issue has yet to be resolved.<\/p>\n

I for one am all for notification
\nwithout excessive nagging.<\/p>\n

The obvious answer to this dilemma is
\nto store the previous state of the server such that it can be used to
\ntest against the currently state; if the state of the server has
\nchanged, a notification gets sent. Thus one straightforward notification that something has changed.<\/p>\n

As a bonus, by reporting on the change
\nof state, the script will now report on when the server has come back
\nonline as well as when it has failed. This simple change eliminates
\nwhat would have been a manual process previously; notifying
\nstakeholders that the issue has been resolved.<\/p>\n

Since the Perl script is evoked by cron<\/a>
\non a regular basis and terminates once polling is complete, the
\n“current” state of a site will need to be store in secondary
\nmemory, i.e. on disk, for future comparison. This is pretty
\nstraightforward in Perl:<\/p>\n

\n
sub logState ($$) {\nmy ( $host, $state, $time ) = @_;\n# Create a filehandle on our log file\nmy $fh = FileHandle->new(\">> $fileLoc\");\nif (defined $fh) {\n# Print to file the necessary information\n# delimited with a colon\nprint $fh \"$host:$state:\" .$time->datetime. \"\\n\";\n$fh->close;\n}\n}\n<\/pre>\n<\/div>\n
With a new Filehandle object the script
\nopens the file previously assigned to the $fileLoc variable for
\nappending (the ‘>>’ immediately prior to the variable denotes
\nwrite by appending).<\/p>\n
If a Filehandle object has been
\nsuccessfully created, the next step is to write a line to the file
\nwith the information necessary for the next iteration of the monitor
\nscript, specifically the host information and its current state.<\/p>\n
Note that each line (\\n) in the file
\nwill denote information about a specific site and that the related
\ninformation is separated by a colon (:). This will be pertinent later in the code, reading of the log file at the next scheduled
\nexecution of the monitor script:<\/p>\n
\n
# Our array of polling sites' previous state\nmy @hostStates = ();\n# Populate said array with information from log file\nmy $fh = FileHandle->new(\"< $fileLoc\");\nwhile ( <$fh> ) {\nmy( $line ) = $_;\nchomp( $line );\npush ( @hostStates, $line );\n}\n$fh->close;\n<\/pre>\n<\/div>\n
In this bit of code the goal is to get
\nthe previously logged state of each site and populate an array with
\nthe information. At the moment how each record is delimited isn’t of
\nconcern, but simply that each line is information relating to a
\nspecific site and gets its own node in the array.<\/p>\n
Note, since the objective here is to simply read the log file the “<” is used by the filehandle to denote that the file is “read-only” and not “append”.<\/p>\n
Once the polling of a specific site
\noccurs, the first item of concern is determining the site’s previous state. For
\nthat the following bit of code is put to use:<\/p>\n
\n
sub getPreviousState ($) {\nmy ( $host ) = @_;\n# For each node in the array do the following\nforeach ( @hostStates ) {\nmy( $line ) = $_;\n# Break up the information\n# using our delimiter, the colon\nmy ($domain, $state, $time) = split(\/:\/, $line, 3);\n# If we find our site return the previous state\nif ( $domain eq $host ) {\nreturn $state;\n}\n}\n}\n<\/pre>\n<\/div>\n
In this function each element in the
\narray is broken down to relevant information using the split
\nfunction<\/a>, which delimits<\/a> the record by a given character, the colon.
\nFrom here it is a simple matter of testing the two states, the
\nprevious and current state before rolling into the notification
\nprocess.<\/p>\n
The complete, improved remote monitor:<\/p>\n
\n
#!\/usr\/bin\/perl\nuse strict;\nuse FileHandle;\nuse Time::Piece;\nuse LWP::UserAgent;\nuse Net::Ping;\nuse Net::Twitter::Lite;\n### Start Global Settings ###\nmy $fileLoc = \"\/var\/log\/state.txt\";\nmy @hosts = ( \"pdw.weinstein.org\", \"www.weinstein.org\" );\n### End Global Settings ###\n# Our array of polling sites' previous state\nmy @hostStates = ();\n# Populate said array with information from log file\nmy $fh = FileHandle->new(\"< $fileLoc\");\nwhile ( <$fh> ) {\nmy( $line ) = $_;\nchomp( $line );\npush ( @hostStates, $line );\n}\n$fh->close;\n# Clear out the file by writing anew\nmy $fh = FileHandle->new(\"> $fileLoc\");\n$fh->close;\nforeach my $host ( @hosts ) {\nmy $previousState = getPreviousState( $host );\nmy $url = \"http:\/\/$host\";\nmy $ua = LWP::UserAgent->new;\nmy $response = $ua->get( $url );\nmy $currentState = $response->code;\nmy $time = localtime;\n# If states are not equal we need to notify someone\nif ( $previousState ne $currentState ) {\n# Do we have an status code?\nif ( $response->code ) {\nreportError(\"$host reports\n$response->message.\\n\");\n} else {\n# HTTP is not responding,\n# is it the network connection down?\nmy $p = Net::Ping->new(\"icmp\");\nif ( $p->ping( $host, 2 )) {\nreportError ( \"$host is responding,\nbut Apache is not.\\n\" );\n} else {\nreportError ( \"$host is unreachable.\\n\" );\n}\n}\n# Not done yet, we need to log\n# the current state for future use\nlogState( $host, $currentState, $time )\n}\nsub reportError ($) {\nmy ( $msg ) = @_;\nmy $nt = Net::Twitter::Lite->new(\nusername => $username,\npassword => $pasword );\nmy $result = eval { $nt->update( $msg ) };\nif ( !$result ) {\n# Twitter has failed us,\n# need to get the word out still...\nsmsEmail ( $msg );\n}\n}\nsub smsEmail ($) {\nmy ( $msg ) = @_;\nmy $to = \"7735551234\\@txt.exaple.org\";\nmy $from = \"pdw\\@weinstein.org\";\nmy $subject = \"Service Notification\";\nmy $sendmail = '\/usr\/lib\/sendmail';\nopen(MAIL, \"|$sendmail -oi -t\");\nprint MAIL \"From: $from\\n\";\nprint MAIL \"To: $to\\n\";\nprint MAIL \"Subject: $subject\\n\\n\";\nprint MAIL $msg;\nclose( MAIL );\n}\nsub logState ($$) {\nmy ( $host, $state, $time ) = @_;\n# Create a filehandle on our log file\nmy $fh = FileHandle->new(\">> $fileLoc\");\nif (defined $fh) {\n# Print to file the necessary information,\n# delimited with a colon\nprint $fh \"$host:$state:\" .$time->datetime. \"\\n\";\n$fh->close;\n}\n}\nsub getPreviousState ($) {\nmy ( $host ) = @_;\n# For each node in the array do the following\nforeach ( @hostStates ) {\nmy( $line ) = $_;\n# Break up the information using our delimiter,\n# the colon\nmy ($domain, $state, $time) = split(\/:\/, $line, 3);\n# If we find our site return the previous state\nif ( $domain eq $host ) {\nreturn $state;\n}\n}\n}\n<\/pre>\n<\/div>\n
\"\"