How can I automatically restart my Docker containers with container auto-delete?

I have a general purpose VPS, and am Dockerising the apps on it. There will be around 5-6 containers on it, and very little else, so the box can be trivially rebuilt as required.

For each app I have a start script. A WordPress container looks like this:

#!/bin/bash

# Get the host IP address
export DOCKER_HOSTIP=`ifconfig docker0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`
echo "Connecting to database on Docker host ${DOCKER_HOSTIP}"

docker run \
    --add-host=docker:${DOCKER_HOSTIP} \
    --network dockernet \
    --network-alias jonblog \
    --detach \
    --restart always \
    --rm \
    jonblog

However, that returns an error:

Conflicting options: --restart and --rm

There are several Docker tickets that say this is sensible, but I don't understand it. I think my meaning is clear: if a container is not running (e.g. on start-up) then I want to start it. If it dies then I expect the container to be removed, and a new fresh one created from the base image. Containers should be immutable anyway - any state I wish to preserve, like media files and logs, will be written to volumes.

So, I thought I should drop the --restart flag, and then use a process manager to stop and start Docker containers. Could I use Monit here? I was hoping to be able to do something like:

CHECK PROCESS jonblog MATCHING jonblog
  START PROGRAM = "/root/docker/jonblog/host-start.sh"
  STOP PROGRAM = "docker stop jon-blog"

However, that checks the system process table, and not docker ps, and so it won't find something matching the specified string. Can I get it to exec docker ps periodically, and match lines in the output?

I'd be happy to use another tool if it proves robust. For example, I find Supervisor a bit heavyweight, but if that is more able to work with Docker, I am willing to use it.

Clarification on --rm

The reason why I want --rm is that during the Dockerisation process, I stop the currently running container, load a new version of the image, and re-run the above script. This means that Docker is being notified of a restart policy for each container. I found that after rebooting the box, I would have 15 or so slightly-differing versions of the app running simultaneously, which is not the intention.

I suppose I could use docker update --restart never on old containers to prevent this happening, but then when my container stops, I am left with it lying around, and I'd just as soon have it auto-delete. I could periodically clean old ones up using some sort of cron job, but that feels a bit hacky given that Docker can do it for me.

Seeking a range of answers

A very helpful comment has suggested I should look into Minikube, which apparently simplifies the setup of Kubernetes, even to the degree a suitably skilled individual can be up and running in five minutes.

I would still quite like to see more lightweight solutions proposed, so I have a range of answers to choose from. As indicated, I would like to find out the answer to whether a process supervisor like Monit would work.

Off the top off my head, I could write a shell loop to write docker ps to a file every five seconds for a minute, and then run that on a Cron every minute. I could then scan that file using grep and the Monit CHECK PROGRAM system check. That's a bit hacky, but is something that I can understand easily if there is a problem with it. Any advances on that suggestion?

Solution 1:

I have an answer that is suitable for my current understanding of Docker. I was advised in the comments to try Minikube, and although undoubtedly this can be spun up quickly, I feared that this would be a rabbit-hole of learning that would get me stuck in tar for weeks. One of my engineering principles is to know when one has reached a cognitive limit for stuffing in new information!

Thus, I set out to resolve this problem in a simple fashion. I had two choices:

1. Use the container auto-delete feature in Docker, and set up my own restart system
2. Use the Docker restart policy, and set up my own container deletion system

I started on the first of these, with the idea that the process supervisor Monit would be nice to use, partly because it is lightweight, and partly because I am familiar with it. However, it started to feel like the wrong solution, since I'd be working around the core problem that it cannot cleanly get a Docker container process list.

In fact, the second option was much cleaner, and this was amplified by the fact that stopped container clean-up is not actually a priority - it is just to keep things tidy. Of course, I used Docker for this; here's the Dockerfile:

# Docker build script for Docker Tidy

FROM alpine:3.6

RUN apk update
RUN apk add docker

# See this for BusyBox cron schedules
# https://gist.github.com/andyshinn/3ae01fa13cb64c9d36e7
COPY bin/docker-tidy.sh /etc/periodic/daily/
RUN chmod +x /etc/periodic/daily/docker-tidy.sh

# Start Cron in the foreground
ENTRYPOINT ["crond", "-l", "2", "-f"]

And here's bin/docker-tidy.sh:

#!/bin/sh
#
# With thanks to:
# http://www.doublecloud.org/2015/05/simple-script-to-list-and-remove-all-stopped-docker-containers/

docker rm -v $(docker ps -a -q -f status=exited)

Finally, one drawback with my solution is that if the host is rebooted prior to a stopped container cleanup, those containers seem to restart as well. I therefore reset the restart policy on those containers prior to starting new ones.

For example, here is how I start the Docker Tidy container itself, on the host. In practice I've tidied up the policy change code into its own script, but this will give the general idea:

#!/bin/bash

# Removes the restart policy from previous containers
CONTAINER_LABEL=docker-tidy-instance
docker ps --all --filter label=$CONTAINER_LABEL --quiet | xargs --no-run-if-empty docker update --restart no

docker run \
    --label $CONTAINER_LABEL \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --detach \
    --restart always \
    docker-tidy