How to get Apache2 to wait for network initialization?
Server just upgraded from Squeeze to Wheezy. After reboots, I noted that apache2 wasn't starting, so installed bootlogd to inspect the output of init scripts.
The error output from apache start was,
Tue Feb 3 08:49:55 2015: [....] Starting web server: apache2(99)Cannot assign requested address: make_sock: could not bind to address [0123:4567:890:abc::d]:80
Tue Feb 3 08:49:55 2015: no listening sockets available, shutting down
Tue Feb 3 08:49:55 2015: Unable to open logs
Tue Feb 3 08:49:55 2015: Action 'start' failed.
Tue Feb 3 08:49:55 2015: The Apache error log may have more information.
Timestamps in the boot output show 4s between "Configuring network interfaces" (08:49:51) and starting apache.
I've worked around this by inserting a sleep 5
into /etc/init.d/apache2
after which apache2 starts fine, so I'm reading it as slow network initialization (possibly specific to IPv6 networking). The server is running on VMWare, and I have access only to the VM.
- What better methods could I use to make Apache wait a few seconds and check its addresses are up and running before start?
- Or to make the network init script not exit until the addresses are ready?
What better methods could I use to make Apache wait a few seconds and check its addresses are up and running before start?
Well, the method I am outlining obviously does not solve the underlying issue. But I believe using a stand-alone monitoring tool that can handle the “kicking Apache into service” task without having to hack the init.d
script is a more stable solution. The key is to use monit
which is self-described as:
Monit is a small Open Source utility for managing and monitoring Unix systems. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.
To install it in Ubuntu, just do this; I am using aptitude
and this is on Ubuntu 12.04 FWIW:
sudo aptitude install monit
Once installed I like to setup the default mail server for sending out alerts. This assumes you have postfix
or sendmail
active on your server. Open up the monit
control file with your favorite text editor; I like to use nano
:
sudo nano /etc/monit/monitrc
Look for the set mailserver
lines and set this line and save the monit
control file:
set mailserver localhost
Now check and make sure there is a monit
conf.d
directory setup; this is where individual monit
tasks are setup:
ls -la /etc/monit/conf.d
If somehow that /etc/monit/conf.d
was not set up, create it like this:
sudo mkdir /etc/monit/conf.d
Now with that done, let’s create a monit
Apache ruleset. First, figure out where the Apache .pid
file is set and check that it is set. This is usually where it is set on Ubuntu 12.04 using the default Apache package install:
ls -la /var/run/apache2.pid
If that .pid
filepath is correct, let’s create the actual monit
Apache ruleset with nano
like this:
sudo nano /etc/monit/conf.d/apache2.conf
And place this code in that file and save it:
check process apache with pidfile /var/run/apache2.pid
start "/etc/init.d/apache2 start"
stop "/etc/init.d/apache2 stop"
if failed host 127.0.0.1 port 80
with timeout 15 seconds
then restart
alert [email protected] only on { timeout, nonexist }
The logic to the script is fairly simple: The monit
Apache ruleset will check the /var/run/apache2.pid
file and it knows to use specific init.d
directives for start
and stop
logic. The magic comes from the if
/then
block which basically monitors Apache port 80
on localhost 127.0.0.1
and will take action to recover Apache if there is a timeout of 15 seconds or more. And the alert
line will send e-mail alerts to a specified e-mail address if the conditions of timeout
or nonexist
are met; that is optional so feel free to comment that out if you don’t have the need to be flooded by e-mail alerts.
Now restart monit
:
sudo service monit restart
And you can follow the monit
log here to see it do it’s thing and debug it if something doesn’t work as expected:
sudo tail -f -n 200 /var/log/monit.log
So when the dust settles on that server, you will have monit
setup to make sure Apache is running. Very useful as a general “keep the Apache web server alive” tool, but in your case monit
can take over the task of making sure Apache comes up on reboot/startup so you don’t have to muck around with core Apache init.d
script which is really a bit messy and is easily forgotten in the future during an upgrade.
Also note, I can see that you are binding Apache explicitly to IPv6, but this monitoring script assumes that if it can’t reach 127.0.0.1
then it should take action. Maybe you need to change the IPv4 localhost address of 127.0.0.1
to the IPv6 equivalent of address of ::1
. Experiment and see what works.