how to avoid timeouts during server reboot
As an ad hoc workaround (only in the regretful absence of a loadbalancer), you could accompany the server reboot with suitable configurations at one of its next hops (a switch, a router, a firewall), e.g.,
- stop apache
- Make the next hop respond with explicit "not reachable" messages
- reboot the server
- Undo changes made step 2
Specifically, step 2 might consist of one of the following
- adding the web server ip as secondary ip address to one of its interfaces (provided there is no web server running on it)
- or create a static arp entry (i.e., ip to mac translation) that refers to a runnnng host that does not run a webserver
- or make that single ip non-routable
- or add a firewall rule to reject (not drop!) port 80 traffic to the ip in question
- or modify an existing SNAT entry for your webserver to refer to a host that is running but not a webserver
- or attach a dummy host with the same MAC address to the same lan
Possibly other variants exist as well. There is of course one important point: Don't shoot yourself in the foot! For example,
- You might accidentally break other things when introducing such misconfigurations
- You might forget to undo the changes
- your server might not be able to properly close existing connections
- Your rebooted server might be unhappy if it cannot see the internet immediately due to this method (it might desperately want to check for updates, or licences, or time, or ...)
Also, i did not test any of these methods myself, I just came up with them and think they should work.
Use a Loadbalancer Software with healthcheck like Zen-Loadbalancer or HA-Proxy.
There you can configure a healthcheck which pings or open a file on your Webserver and remove them from the Load-Balancing if the file is not available for example.
DNS Round-robin is not made for such things because your DNS doesn't know if a server is not available. The DNS give the requests alternately to every ip.
Just to explain: DNS Round Robin is not designed for the purpose you are using it for. It simply tells the DNS to hand out the available IP-addresses in an alternating manner; there is no feedback between the failing server and the DNS (and if there were, timing problems would prevent it to be of much help, because of the way the DNS is designed). So you can use DNS-Round-Robin for loadbalancing (in a limited way) but not for failover.
See this quote: "If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service." from http://en.wikipedia.org/wiki/Round-robin_DNS
The reason you did not have any problems with the webserver shut down, is probably, that your browser sticks to the webserver that works, maybe because of some session cookie or the like. You should try your experiment with the web server shut down several times, using completely new sessions - you should experience the same problems as with reboots then.
Besides from that I would second the answer already given: a loadbalancer solution would do what you want.