How do I configure my Upstart scripts to take dependencies into account?

I have install PostgreSQL, Atlassian JIRA, Stash, Confluence and Crowd on a single Ubuntu 12.04 Server. They are integrated and working per Atlassian documentation.

One thing I did notice is that the product suite doesn't tolerate dependent components being down. For example, if Crowd is stopped, users can no longer login. If PostgreSQL is being stopped for maintenance reasons, the dependent applications may never recover.

Every night the system goes through a maintenance window, during which the operating system and software may be patched. The external DNS is configured to gracefully show a maintenance page if there is an error with the applications.

Every now and then, PostgreSQL may be stopped, patched and then started. How do I configure my Upstart scripts to take dependencies into account? I configured the "start on", but it has no effect.

I'm comfortable writing Windows services and configuring their dependencies during install (e.g. windows installer, or PowerShell).

Testcases:

• I stopped all services. I then started Confluence, expecting that it would either fail (since PostgreSQL and Crowd is stopped), or would automatically attempt to start PostgreSQL and Crowd. Neither happened.

• I started all services. I then stopped Crowd, expecting that Confluence, JIRA and Stash will be stopped. It only stopped Crowd and all the other services kept running, but failed to function correctly.

Dependencies:

• Crowd: PostgreSQL, Networking
• Confluence: PostgreSQL, Networking, Crowd
• JIRA: PostgreSQL, Networking, Crowd
• Stash: PostgreSQL, Networking, Crowd

Confluence

# confluence

description "Atlassian Confluence"

start on (net-device-up and local-filesystems and postgresql and runlevel [2345] and crowd)
stop on runlevel [!2345]

respawn

kill timeout 30

env RUN_AS_USER=confluence
env BASEDIR=/usr/local/bin/confluence

script
    LOGFILE=/var/log/confluence/confluence.`date +%Y-%m-%d`.log
    exec su - $RUN_AS_USER -c "$BASEDIR/bin/catalina.sh run" >> $LOGFILE 2>&1
end script

JIRA

description "Atlassian JIRA"

start on (net-device-up and local-filesystems and postgresql and runlevel [2345] and crowd)
stop on runlevel [!2345]

respawn

kill timeout 30

env RUN_AS_USER=jira
env BASEDIR=/usr/local/bin/jira

script
    LOGFILE=/var/log/jira/jira.`date +%Y-%m-%d`.log
    exec su - $RUN_AS_USER -c "$BASEDIR/bin/catalina.sh run" >> $LOGFILE 2>&1
end script

Stash

description "Atlassian Stash"

start on (net-device-up and local-filesystems and postgresql and runlevel [2345] and crowd)
stop on runlevel [!2345]

respawn

kill timeout 30

env RUN_AS_USER=stash
env BASEDIR=/usr/local/bin/stash
env STASH_HOME="/var/local/lib/stash"

script
    LOGFILE=/var/log/stash/stash.`date +%Y-%m-%d`.log
    exec su - $RUN_AS_USER -c "$BASEDIR/bin/start-stash.sh -fg" >> $LOGFILE 2>&1
end script

Crowd

# crowd

description "Atlassian Crowd - Single Sign-On (SSO) and Identity Management"

start on (net-device-up and local-filesystems and postgresql and runlevel [2345])
stop on runlevel [!2345]

respawn

kill timeout 30

env RUN_AS_USER=crowd
env BASEDIR=/usr/local/bin/crowd/apache-tomcat

script
    LOGFILE=/var/log/crowd/crowd.`date +%Y-%m-%d`.log
    exec su - $RUN_AS_USER -c "$BASEDIR/bin/catalina.sh run" >> $LOGFILE 2>&1
end script

PostgreSQL was installed via APT and starts up automatically during startup.

Try:

• Crowd

  start on (net-device-up and started postgresql)
  stop on stopping postgresql
  

• Confluence

  start on started crowd
  stop on stopping crowd
  

• Stash

  start on started crowd
  stop on stopping crowd
  

• JIRA

  start on started crowd
  stop on stopping crowd
  

Reference: http://upstart.ubuntu.com/cookbook/

• 6.32.2 Start depends on another service
• 6.33.2 Stop before depended-upon service

This is going to be a little hard, because postgresql is not a native upstart job. You need to do the following:

1. Add the line initctl emit -n started JOB=postgresql at the END of the start portion of /etc/init.d/postgresql
2. Add the line initctl emit stopping JOB=postgresql to the BEGINNING of the stop portion of /etc/init.d/postgresql

3. Change crowd to:

   start on started postgresql
   stop on stopping postgresql
   

That should be it. If you need help for steps 1 and 2, just ask.