How can I crontably reboot a Linux host upon network failure? [closed]

I have a personal Ubuntu host, connected to a public/sharing Wi-Fi AP (not under my control).

Sometimes, it would have a network issue, and I am very sure, only restarting the network service would not work. The only way is to reboot it.

My plan is to add a crontab, to test the network connection. If it fails, then reboot the computer.

If I manually run the auto_reboot.sh, it does reboot, when a ping test fail. But running from crontab, it is not working:)

Here is my crontab entry

crontab -l
* * * * * /root/loadrc/transmissionrc/auto_reboot.sh

File /root/loadrc/transmissionrc/auto_reboot.sh

#!/bin/zsh

/root/loadrc/networkrc/ping.sh
rc=$?

if [[ $rc -eq 0 ]]
then
    echo "say The internet is back up."
else
    reboot
fi

File /root/loadrc/networkrc/ping.sh

#!/bin/zsh
((count = 10))                           # Maximum number to try.

while [[ $count -ne 0 ]] ; do
    ping -c 1 8.8.8.8                    # Try once.
    rc=$?
    if [[ $rc -eq 0 ]] ; then
        ((count = 1))                    # If okay, flag loop exit.
    else
        sleep 1                          # Minimise network storm.
    fi
    ((count = count - 1))                # So we don't go forever.
done

exit $rc

I add some logs, and deliberately bring down the Wi-Fi interface:

watch ifconfig wlan0 down

transmissionrc/auto_reboot.sh
#!/bin/zsh

echo "" > /root/loadrc/crontab.log

/root/loadrc/networkrc/ping.sh
rc=$?

if [[ $rc -eq 0 ]]
then
    echo "say The internet is back up."
else
    reboot
fi

networkrc/ping.sh
#!/bin/zsh
((count = 10))                           # Maximum number to try.


while [[ $count -ne 0 ]] ; do
    /usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1
    echo "step --> 2" >> /root/loadrc/crontab.log
    rc=$?

    if [[ $rc -eq 0 ]] ; then
        echo "step --> 3" >> /root/loadrc/crontab.log
        ((count = 1))                    # If okay, flag loop exit.
    else
        echo "step --> 4" >> /root/loadrc/crontab.log
        sleep 1                          # Minimise network storm.
    fi
    ((count = count - 1))                # So we don't go forever.
done

exit $rc

File /root/loadrc/crontab.log

/usr/bin/ping: connect: Network is unreachable
step --> 2
step --> 3

which means, in the crontab mode, even the ping test fail, the return code is still zero.

So the question comes to: how can I test the network connection in crontab mode?

Solution 1:

I have three thoughts on possible solutions to your issue:

1. You said:

If I manually run the auto_reboot.sh, it does reboot, when a ping test fail. But running from crontab, it is not working:)

Usually, when a command runs properly in your interactive shell (from the CLI), but fails to run properly under cron it is due to a difference in the environment; e.g. cron has a different PATH than you do from your interactive shell. Typically, the cron environment is: PATH=/usr/bin:/bin. Any script you run run under cron will not be able to find executables that are not on the PATH.

As an aside, you can check the cron environment on your system by simply running env using your crontab:

* * * * * /usr/bin/env > /my/cronlog/location/mycronenvironment.txt 2>&1

In your auto_reboot.sh, you failed to use a full path specifications for reboot. As reboot is typically found in /sbin/reboot, and /sbin may not be in the PATH used by cron, this is a potential problem.

Consequently, I will suggest that you verify the environment (PATH) used by cron, and double-check all of your commands are either: 1) on the cron PATH, or 2) use a full path specification.

2. You run everything out of the /root directory

Ordinarily, /root isn't used for user scripts. Perhaps you are using sudo? Or, perhaps you have done an su to become root? If this is the case, I would comment that this is not best practice, even though it can still work. I feel best practice is to use sudo from your user account for any privilege escalation you need.

Without trying to be pedantic, I'd like to say that the root account has a crontab that runs independently of any user crontab. Also, the root crontab does not require sudo be used - everything done in the root crontab is done with root privileges.

All of that said, I see the call to reboot in your script - a command that requires root privileges to run. This will work as you've written it only when used in the root crontab. Your question did not indicate whether or not you are using su or sudo, and so I went through this in an effort to make two points clear:

1. If your cron job requires root privileges, it may be best to run that job from the root crontab. The alternative is to use sudo in a user crontab which is potentially awkward if authentication is required for sudo - as is often the case.

2. The root crontab may be accessed from a regular user account simply by using sudo crontab -e; i.e. one is not required to su to root to access the root crontab.

3. You may have a logical error in your script

As pointed out in another answer, it is not clear that you can rely on the value of rc from your ping.sh script as a condition for reboot. Unfortunately, whether or not this is an issue is masked by what seems to be two different versions of the ping.sh script in your question - it is unclear whether you are using the first version:

#!/bin/zsh
((count = 10))                           # Maximum number to try.

while [[ $count -ne 0 ]] ; do
    ping -c 1 8.8.8.8                    # Try once.
    rc=$?

or the second version:

#!/bin/zsh
((count = 10))                           # Maximum number to try.


while [[ $count -ne 0 ]] ; do
    /usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1
    echo "step --> 2" >> /root/loadrc/crontab.log
    rc=$?

Strictly as my personal choice, I would favor combining the code from these two scripts (ping.sh and auto_reboot.sh) into a single script because it seems more straightforward to me, but you may have good reasons for doing it this way, and there's no reason it won't work if done correctly.

Solution 2:

/usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1 
echo "step --> 2" >> /root/loadrc/crontab.log  
rc=$?

I think this checks the exit code of the echo command while your logic needs the exit code of the ping command.