How to start etcd in docker from systemd?

Solution 1:

Update: Posting test data and integrating updates based on comments received. docker -d is not required for systemd integration, as originally thought. Type= setting as Michael indicated seems to be in my experience, more important than offloading the daemonized status of a service to docker. The OP problem seemed on first blush to be a side effect of not having backgrounded, as I originally explained. This background seems irrelevant after having tested it further.

Note that the Amazon AWS image used in the OP, is not something that I can test or directly troubleshoot. A contrasting example for etcd and systemd are shown here to help with configuring the endpoint system similarly to mine. System details:

• Ubuntu 20.04 LTS
• docker 20.10.7
• etcd 3.5.0

systemd configuration

I ended up with the following systemd service file. Note that Type=simple, owing to Michael suggesting to clarify this point in the response (and, apparently, my own understanding of this piece of the puzzle). You can learn more about systemd types here:

https://www.freedesktop.org/software/systemd/man/systemd.service.html

Type matters; More to the point my original understanding of simple as type, was myopically focused on the lack of communication back to systemd, which caused me to ignore the applicable behavior of what the type setting does in reaction to responses from the called application (in this case docker).

Removal of type, or addition of type to simple, will result in the same behavior regardless. The following configuration in my test worked reliably, as did -d being present or not in the docker run command:

[Unit]
Description=Docker container-etcd.service
Documentation=man:docker
Requires=docker.service
Wants=network.target
After=network-online.target

[Service]
ExecStartPre=- /usr/bin/docker stop etcd
ExecStartPre=- /usr/bin/docker rm etcd
ExecStart=docker run --rm -d -p 2379:2379 -p 2380:2380 --volume=/home/user/etcd-data:/etcd-data --name etcd quay.io/coreos/etcd:v3.5.0 /usr/local/bin/etcd --data-dir=/etcd-data --name etcd --initial-advertise-peer-urls http://10.4.4.132:2380 --listen-peer-urls http://0.0.0.0:2380 --advertise-client-urls http://10.4.4.132:2379 --listen-client-urls http://0.0.0.0:2379 --initial-cluster etcd=http://10.4.4.132:2380
ExecStop=/usr/bin/docker stop etcd -t 10
ExecRestart=/usr/bin/docker restart etcd
KillMode=none
RemainAfterExit=1
Restart=on-failure
Type=simple

[Install]
WantedBy=multi-user.target default.target

Notes

• RemainAfterExit added, as systemd will consider the service exited after start if not present; The lack of this boolean creates a seemingly erroneous situation where docker ps shows the container running, but systemctl status container-etcd showed exited and inactive.
• The systemd unit file is somewhat syntactically incorrect. %n is typically used for the Exec lines to refer to the service name (as in ...docker restart %n); I did not want to introduce further confusion while attempting to solve the OP's problem. Not to mention I went with etcd as the docker container name, versus container-etcd, as the unit service name.
• ExecStart was collapsed to a one-line command. \ standard syntax did not work for me, nor did quoting the etcd call command to the container. My tests yesterday seemed to have worked fine, but today's configuration didn't behave the same way as yesterday. So I redid the test and configurations to find what seemed to be most stable for me.
• Obviously, if you're going to be using docker rm at any point, you must or very strongly should use bind mounts, as stated in the OP and here with --volume. Personally I use full path locations, all stored under /srv, and then bind mount into the container. That way I have one folder to backup, and the state of the containers, present or not is irrelevant.

Confirmation

After updating systemd service file, doing a daemon-reload, etc., I execed into the container and ran a test command against etcd:

• docker exec -it etcd sh
• etcdctl --endpoints=http://10.4.4.132:2379 member list

Result

9a552f9b95628384, started, etcd, http://10.4.4.132:2380, http://10.4.4.132:2379, false