Shell script to execute something when one of the daemon dies?

I just had to build two instances of daemon running on different ports. let's say they both serve critical missions for some applications.

How can I do some automatic task (eg. shell script) execution for checking both of daemons when one of them fails to serve?

What kind of script that can always check the daemon's life and maybe can execute some other jobs if one of the daemons incidentally stopped?

Solution 1:

monit is great in this case - it runs on localhost, so you don't need a network connection to restart your daemon (in case it fails, or daemon is responsible for networking). It also has small footprint on the system, and you can use it to monitor your other daemons/disk space/etc. as well.

Create a start/stop script (similar to those in /etc/init.d/ and create symlinks for it in runlevels that your system uses for normal operation, ensuring that your daemon will start at reboot and stop at shutdown properly. If your daemon doesn't have pidfile, create one using start-stop-daemon script.

After that, install monit and create configuration for your daemon, something like this:

check process daemond with pidfile /var/run/daemond.pid
    start program "/etc/init.d/daemond start"
    stop program "/etc/init.d/daemond stop"
    if failed port 1234 type TCP for 5 times within 10 cycles then restart
    if 3 restarts within 5 cycles then alert

This configuration will ensure that if daemon stops responding on tcp port 1234, or will stop running, it will be restarted using the init script. monit will also send you an alert through email, or do other things, depending on how you configure it. Just check out monit(1) manpage.