Why does upstart keep respawning my process?

Solution 1:

The difficulty here is the combination of 'respawn' with a pre-stop script that tells the process to stop. From init(5):

   respawn
         A service or task with this stanza will be automatically started
         if it should stop abnormally.  All reasons for a service stopping,
         except the stop(8) command itself, are considered abnormal.  Tasks
         may exit with a zero exit status to prevent being respawned.

The documentation is a little unclear on the point of whether exiting with a zero exit status should cause a respawn. However, fundamentally you've found an upstart bug because the main process ending when the goal is 'stop' should not result in a change to 'respawn'.

To work around this bug, you should be able to use "normal exit" to tell upstart that this is a normal way to stop the job and that it should not respawn.

  normal exit STATUS|SIGNAL...
         Additional exit statuses or even signals may be added, if the
         job process terminates with any of these it will not be considered
         to have failed and will not be respawned.

         normal exit 0 1 TERM HUP

Note that in general, it would be more robust to kill the process with a signal (specifying "kill signal N" if necessary) instead of with a pre-stop process that issues commands; but of course this is not always possible if the service doesn't support clean shutdown upon receipt of a signal.

Solution 2:

In case anyone else ends up here I've submitted a working patch to the bug:

https://bugs.launchpad.net/upstart/+bug/568288/comments/6