Intermittent "Address already in use: AH00072: make_sock: could not bind to address" on port 80

We are experiencing intermittent "Address already in use: AH00072: make_sock: could not bind to address" on port 80 failures on our servers. I have looked at other answers and they are mostly based around it being a consistent failure and a LISTEN being defined twice, which is not our issue.

Our issue only happens shortly after boot of the instance where we "sudo service httpd restart" the service to load a new config we lay down depending on the environment.

I was able to catch the issue live and see that it was in fact httpd that is bound. The issue appears to be that Apache/httpd is hung and can't restart and then the restart doesn't wait for it.

We are using Apache 2.4 on Amazon Linux 1.

There are two main reasons for getting an Address Already in Use error:

1. Another Server Uses that IP:port

   In this case, you try to start two services and both want the same address & port combo, which is not possible. Only one service can listen on one specific port. Apparently, you are not in this situation.

2. The same Server Gets Restarted too Quickly

   By default, the kernel will hold on an IP:port combo for a few seconds (possibly a minute) before allowing a service to use these again. This is to make sure that all the clients had a chance to receive a proper close signal. Also, for security reasons, you can end up receiving packets that were expected to be sent to the old connection. That can cause issues.

   It looks like that would be your situation, although I thought Apache would automatically use the SO_REUSEADDR flag. The way to fix this is to look at the implementation of the restart and add a sleep between the shutdown and start commands. It's pretty easy if you are using an init script:

    [...snip...]
        ;;
    restart)
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop stop
        case "$?" in
                0|1)
                        sleep 12   # <<-- sleep before restarting
                        do_start
                        case "$?" in
    [...snip...]
   

   With systemd, you'd have to edit the .service file and add a sleep before you start the apache2 server.

    ExecStart=sleep 20 && /usr/sbin/apache2
   

Your initialization order may also be the cause of the issue (if you have to start, change something and then restart apache2 on a boot; that being said, being able to restart apache2 at any point in time is probably important).