How to automatically restart a service on failure in Linux

Solution 1:

monit is a great way to monitor and restart services when they fail--and you'll probably end up using this for other essential services (such as Apache). There's a nice article on nixCraft detailing how to use this for services specifically, although monit itself has many more functions beyond this.

As for the socket aspect, @galraen answered this spot on.

Solution 2:

Only answering the service restart part. I came across Monit as well, but on CentOS 7 systemd takes care of all that for you. You just need to add these two lines to the .service file (if they're not there already):

Restart=always
RestartSec=3

See https://jonarcher.info/2015/08/ensure-systemd-services-restart-on-failure/ for reference.

If you want to create a custom systemd service, it's pretty straightforward to write your own service file. See the example below, for a custom http server.

Start the editor with a new service file:

vim /etc/systemd/system/httpd.service

And add the following content, which you can edit as required:

[Unit]
Description=My httpd Service
After=network.target

[Service]
Type=simple
User=root
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PERLLIB=/perl
ExecStart=/bin/httpd /etc/httpd.conf
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

I want it to start automatically on boot:

systemctl enable httpd

Tell systemd about the changes and start the service:

systemctl daemon-reload
systemctl start httpd

And now you can see the status:

systemctl status httpd

Solution 3:

You can call setsockopt(2) for listening socket with SO_REUSEADDR, so you will be able to bind(2) it again without waiting for expiring all connections. Another possibility: drop connections from kernel. FreeBSD have tcpdrop command for this, don't know about Linux.