home / articles / cgi

Maintaing A Web Server

Maintaining a web server is certainly a demanding chore. There are countless things that can fail, from the AC power coming into the building to the CGI scripts that collect customer orders. Recently, my web server failed due to a power failure. Ironically however, it was offline longer because it was protected by an uninterruptable power supply.

I have a web server located in Marina del Ray, California that I access remotely via SSH (secure shell -- a secure replacement for telnet, rlogin and rcp). Because reliability, automation and full remote access are the highest priority, this machine runs FreeBSD version 2.2.6, a free version of BSD-style Unix. In some ways, it feels more like Sun's Solaris than Linux. Quite simply, if you're impressed by Linux, you'll be amazed by FreeBSD. From installation to setup and everyday maintenance, this operating system really shines; everything has a professional feel to it. Most important in this case, the system logs stored in /var/logs provide enough detail to figure out exactly what is going on "behind the scenes."

The machine itself is protected by Best Power's Fortress 520VA Uninterruptable Power System. This model provides a serial cable and software to monitor the UPS and provide safe shutdown of a computer system. At a few minutes past midnight, the power failed in the computer room. The server logs showed that the UPS kicked in and successfully kept the server running. However, other equipment in the room was not protected, most notibly the ethernet hubs and switches. So, when power was restored an hour later, the ethernet card in my machine didn't reestablish its link correctly. My machine, protected from the power failure by a UPS, was cut off from the outside world. To reestablish its network connection, it was manually reset 21 hours later.

This experience shows that attempting to protect against one point of failure -- the AC power -- left me more volunerable to others. In fact there are many problems that can occur outside the machine that will cause it to lose its network connection. Realizing this, I wrote a simple Perl script to monitor the network and safely reset the machine if it loses contact with the gateway for more than 10 minutes. Of course, if the gateway goes down for some other reason, my machine will be resetting every 10 minutes. But, for all intents and purposes, a web server is useless without its network and connection to the gateway. This may not make sense in all situations, but it's a good quick fix.

The script named pingAlive.pl must be run as root to shutdown and reset the machine. It also requires the Net::Ping Perl module available from CPAN. The script itself has a few options that allow easy customization:

• $host - the remote address to ping. If it doesn't respond in 10 seconds, this is considered an error. In other words, this host is assumed to be very reliable. If it's not available, something is seriously wrong.
• $sleepTime - number of seconds to wait between each ping. This should be set to a reasonable value, somewhere between 30 and 300 seconds.
• $logFile - where the ping failure notices will be written. This way, you can monitor the operation of the script and also determine if it reset your machine.
• $maxErrors - after this number of failed ping attempts, $theCommand will be run.
• $theCommand - this is the command that will be run after the specified number of errors occur. If you want it to reset the machine (using /sbin/shutdown), it is very important the you fully realize the consequences. It may acutually cause more problems. For example, if the network is down and the machine resets, it may then hang during initialization. Be very sure to test this very carefully.

As you can see, this script is a quick hack for an annoying problem. Head on over to the message boards and post your ideas.

This Week's Magical Perl Scripts:

• pingAlive.pl - pings another IP address and reboots if it looses contact.
• showLogs.pl - a quick way to peek at server logs without logging into the machine.

Author: Doug Steinwand
Date: [06/09/98]

More articles about CGI
More articles by Doug Steinwand
Author Biography