control startup order of several servers

You have several options. It may be a good idea to combine two or more of these approaches.

  • If each group is on dedicated UPSs then you can control UPS start-up sequence to some degree. High power UPSs often can be configured to delay their start-up. Stagger the start-up delays to meet your needs. You should be staggering start-up anyway to prevent start=up load from triggering fuses or breakers on your power feed.

  • As other have noted there are PDUs with delay capabilities. These would be configured as for UPSs. They may also be network controllable so the next group can be turned on when the required services are available.

  • WoL is one approach you could use as other have noted.

  • If you are using a single threaded init process, you could add an init script that waits for the required service to be available before proceeding. Alternatively, you could add the checks to the appropriate init scripts. Adding the guard checks for necessary services may be a good idea anyway.

  • You could plumb your Internet facing IP addresses. but not enable them until all the required service are in place. This would require a guard script that verifies the required services are available.

  • NFS mounts can be configured to block until the mounts are available. This should delay further init processing until the NFS servers are serving the required mounts.


I'd question the specific scenario you're planning around...

Are you trying to plan for power-up following a power outage? Recovery from some sort of disaster? Or is your concern more power-spike related? For the latter, I've sometimes used random startup delays configured in the servers' BIOS or used a switched PDU to handle the power-on sequence to prevent overloading.

On the sequencing side, I'd engineer around the dependencies at the application level. From a cold-start, your application servers should be able to tolerate the failure, delay or missing presence of support severs (DHCP/DNS/LDAP). Do you have backup servers running those services? Anything outside of the location?

If not, you could set application or daemon startup checks - e.g. don't start the NFS daemon if the DNS servers can't be reached. I've done a teeny bit of this dependency checking using Monit or a wrapper script... but really haven't given thought to this type of ordering in most environments.


What are you trying to accomplish? Is it minimizing (smoothing out) the spikes in demand created by a bunch of servers simultaneously booting or something that can push emergency adaptation of poweroff? If it's the former then look for power sequencers - there are in-rack units designed to power up groups of outlets with programmable delays - say a few minutes between each of your groups to allow each to settle before booting. This used to be pretty common with some vendors in the larger side of midrange.