Upstart tracking wrong PID of process - not respawning

Solution 1:

Quite late, but hopefully this can be of help to other users.

There is a documented bug in upstart which can cause initctl to track the wrong PID if you specify the incorrect fork stanza in an upstart config: https://bugs.launchpad.net/upstart/+bug/406397

What happens is that upstart checks the fork stanza and determines how many forked processes it should check before choosing the "true" PID of the program being controlled. If you specify expect fork or expect daemon but your program does not fork a sufficient number of times, start will hang. If, on the other hand, your process forks too many times, initctl will track the wrong PID. Theoretically, it should be documented in this section of the upstart cookbook, but as you can see in this situation there is a PID associated with the killed process when there shouldn't be.

The implications of this are explained in the bugtracker comments, but I'll summarize here: besides initctl not being able to stop the daemon process and being stuck in an undocumented/illegal state <service> start/killed, process <pid>, if the process belonging to that PID stops (and it usually will) then the PID is freed up for re-use by the system.

If you issue initctl stop <service> or service <service> stop, initctl will kill that PID the next time it appears. This means that, somewhere down the road if you don't reboot after making this mistake, the next process to use that PID will be immediately killed by initctl even though it won't be the daemon. It could be something as simple as cat or as complex as ffmpeg, and you'd have a hard time figuring out why your software package crashed in the middle of some routine operation.

So, the issue is that you specified the wrong expect option for the number of forks your daemon process actually makes. They say there is an upstart rewrite that addresses this issue, but as of upstart 1.8 (latest Ubuntu 13.04/January 2014) the issue is still present.

Since you used expect daemon and ended up with this issue, I recommend trying expect fork.

Edit: Here's a Ubuntu BASH-compatible script (original by Wade Fitzpatrick modified to use Ubuntu sleep) that spawns processes until the available process ID address space is exhausted, at which point it starts back at 0 and works its way up to the "stuck" PID. A process is then spawned at the PID initctl is hung up on, and initctl kills it and resets.

#!/bin/bash

# usage: sh /tmp/upstart_fix.sh <pid>

sleep 0.001 &
firstPID=$!
#first lets exhaust the space
while (( $! >= $firstPID ))
do
    sleep 0.001 &
done

# [ will use testPID itself, we want to use the next pid
declare -i testPID
testPID=$(($1 - 1))
while (( $! < $testPID ))
do
    sleep 0.001 &
done

# fork a background process then die so init reaps its pid
sleep 3 &
echo "Init will reap PID=$!"
kill -9 $$
# EOF

Solution 2:

For the provided example:

$ initctl status bluepill
bluepill start/running, process 990

a quick solution for me is:

# If upstart gets stuck for some job in stop/killed state
export PID=990
cd /usr/local/bin
wget https://raw.github.com/ion1/workaround-upstart-snafu/master/workaround-upstart-snafu
chmod +x workaround-upstart-snafu
./workaround-upstart-snafu $PID

source: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=582745#37

I hope this will be helpful. What is going on is explained in the other answers.