Force Linux servers to automatically try to get an IP address if they lose it

We use static DHCP leases for our Linux servers on our small business network. Recently something bad happened to the DHCP server and all the linux servers ended up losing their IP addresses since they couldn't find a DHCP server on the network.

Fast forward a day later, the DHCP server was back up and running, but the servers didn't seem to be looking for the DHCP server anymore, so therefore they didn't have an IP address and weren't trying to get one. I ended up having to manually and physically goto each server and restart the network, at which point they would immediately get their IP address back.

Is there anything I can do on the servers to prevent this problem in the future? I'm sure after the DHCP server went down, the servers maybe tried to get an IP address renewed for a while, but eventually they apparently just stopped trying.

Obviously a solution is to have a backup DHCP server, but let's just pretend that isn't an option. Is there some way to force servers to keep trying to renew their IP indefinitely?

The servers are a mix of CentOS and Fedora.

Did they revert to APIPA addressing? 169.254.0.0/16? You can disable that with the NOZEROCONF=Yes directive in /etc/sysconfig/network.

You should also extend your DHCP lease to be twice the longest reasonable outage interval of the DHCP server. Clients will generally wait half of the lease period before trying to renew; if they only check in once every couple of days, you have considerable leeway to fix the server or stand up a new one.

We run fully-reserved DHCP in production; it has not been a reliability problem. In fact, I contend that static IP addressing is one of the last manual operations that folks cling to, and it's largely an anachronism. I've not done a formal analysis, but in three years we've had one DHCP related issue, and that was an admin fat-finger which misfired a "release now" script. On the histogram of "Oops" causes, DHCP is way out on the long-tail.

I know everyone loves to slag on Windows, but IME when APIPA is disabled and DHCP enabled, Windows machines are downright tenacious about holding onto whatever their last DHCP assignment was. I've powered up machines after months of mothball, and wireshark shows them asking for the last DHCP-assigned address.

I use static IP-addresses for servers, DHCP for personal computers only.

Whilst I believe DHCP makes some effort to avoid changing a devices IP-address, this is not quaranteed - at the expiry of a lease the computer might be allocated a different IP-address by DHCP. I've always assumed this would be a bad thing to let happen to a server in the middle of a workday.

You can set PERSISTENT_DHCLIENT=yes in /etc/sysconfig/network-scripts/ifcfg-ethX. This effectively removes the -1 flag that is sent to dhclient by default on RedHat style systems. From man dhclient:

The -1 flag cause dhclient to try once to get a lease. If it fails, dhclient exits with exit code two.

However, in the case that you have a power outage and your servers come back online before your switches do, this does not work due to https://bugzilla.redhat.com/show_bug.cgi?id=234075. Basically that bug says that ifup first check to see if a link is present on the interface before attempting to run dhclient. No link -> no dhclient.

My solution is to add this to root's crontab:

*/5 * * * * ifconfig eth0 | grep -q "inet addr:" || ifup eth0 2>&1 >/dev/null

Also add this to /etc/dhcp/dhclient.conf for speedy recovery:

retry 10
timeout 10

Lease times are set by the server, not by the client. You could try increasing the lease length for whatever scope your servers are assigned to, but that's really just a band-aid. You really need to either set up a redundant DHCP server or preferrably, moving your servers to statically-assigned addresses.