Start N processes with one systemd service file

I found this systemd service file to start autossh to keep up a ssh tunnel: https://gist.github.com/thomasfr/9707568

[Unit]
Description=Keeps a tunnel to 'remote.example.com' open
After=network.target

[Service]
User=autossh
# -p [PORT]
# -l [user]
# -M 0 --> no monitoring
# -N Just open the connection and do nothing (not interactive)
# LOCALPORT:IP_ON_EXAMPLE_COM:PORT_ON_EXAMPLE_COM
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -p 22 -l autossh remote.example.com -L 7474:127.0.0.1:7474 -i /home/autossh/.ssh/id_rsa

[Install]
WantedBy=multi-user.target

Is there a way to configure systemd to start several tunnels in one service.

I don't want to create N system service files, since I want to avoid copy+paste.

All service files would be identical except "remote.example.com" would be replace with other host names.

1.5 year later ...

I asked this question roughly 1.5 year ago.

My mind has changed. Yes, it's nice, that you can do this with systemd, but I will use configuration-management in the future.

Why should systemd implement a template language and substitute %h? .. I think it makes no sense.

Several months later I think this looping and templating should be solved on a different level. I would use Ansible or TerraForm for this now.


Well, assuming that the only thing changing per unit file is the remote.example.com part, you can use an Instantiated Service.

From the systemd.unit man page:

Optionally, units may be instantiated from a template file at runtime. This allows creation of multiple units from a single configuration file. If systemd looks for a unit configuration file, it will first search for the literal unit name in the file system. If that yields no success and the unit name contains an "@" character, systemd will look for a unit template that shares the same name but with the instance string (i.e. the part between the "@" character and the suffix) removed. Example: if a service [email protected] is requested and no file by that name is found, systemd will look for [email protected] and instantiate a service from that configuration file if it is found.

Basically, you create a single unit file, which contains a variable (usually %i) where the differences occur and then they get linked when you "enable" that service.

For example, I have a unit file called /etc/systemd/system/[email protected] that looks like this:

[Unit]
Description=AutoSSH service for ServiceABC on %i
After=network.target

[Service]
Environment=AUTOSSH_GATETIME=30 AUTOSSH_LOGFILE=/var/log/autossh/%i.log AUTOSSH_PIDFILE=/var/run/autossh.%i.pid
PIDFile=/var/run/autossh.%i.pid
#Type=forking
ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i

[Install]
WantedBy=multi-user.target

Which I've then enabled

[user@anotherhost ~]$ sudo systemctl enable [email protected]
ln -s '/etc/systemd/system/[email protected]' '/etc/systemd/system/multi-user.target.wants/[email protected]'

And can intereact with

[user@anotherhost ~]$ sudo systemctl start [email protected]
[user@anotherhost ~]$ sudo systemctl status [email protected]
[email protected] - AutoSSH service for ServiceABC on somehost.example
   Loaded: loaded (/etc/systemd/system/[email protected]; enabled)
   Active: active (running) since Tue 2015-10-20 13:19:01 EDT; 17s ago
 Main PID: 32524 (autossh)
   CGroup: /system.slice/system-autossh.slice/[email protected]
           ├─32524 /usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com
           └─32525 /usr/bin/ssh -L 40000:127.0.0.1:40000 -R 40000:127.0.0.1:40001 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
[user@anotherhost ~]$ sudo systemctl status [email protected]
[user@anotherhost ~]$ sudo systemctl status [email protected]
[email protected] - AutoSSH service for ServiceABC on somehost.example.com
   Loaded: loaded (/etc/systemd/system/[email protected]; enabled)
   Active: inactive (dead) since Tue 2015-10-20 13:24:10 EDT; 2s ago
  Process: 32524 ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i (code=exited, status=0/SUCCESS)
 Main PID: 32524 (code=exited, status=0/SUCCESS)

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopping AutoSSH service for ServiceABC on somehost.example.com...
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopped AutoSSH service for ServiceABC on somehost.example.com.

As you can see, all instances of %i in the unit file get replaced with somehost.example.com.

There's a bunch more specifiers that you can use in a unit file though, but I find %i to work best in cases like this.


Here is a python example, which was what I was looking for. The @ in the service filename lets you start N processes:

$ cat /etc/systemd/system/[email protected]

[Unit]
Description=manages my worker service, instance %i
After=multi-user.target

[Service]
PermissionsStartOnly=true
Type=idle
User=root
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
Restart=always
TimeoutStartSec=10
RestartSec=10

Various methods to call it

Enabling various counts for example:

  • Enable 30 workers:

    sudo systemctl enable my-worker\@{1..30}.service
    
  • Enable 2 workers:

    sudo systemctl enable my-worker\@{1..2}.service
    

Then be sure to reload:

sudo systemctl daemon-reload

Now you can start/stop then in various ways:

  • Start 1:

    sudo systemctl start [email protected]
    
  • Start Multiple:

    sudo systemctl start my-worker@{1..2}
    
  • Stop Multiple:

    sudo systemctl stop my-worker@{1..2}
    
  • Check status:

    sudo systemctl status my-worker@1
    

UPDATE: To manage instances as one service, you can do something like this:

/etc/systemd/system/[email protected]:

[Unit]
Description=manage worker instances as a service, instance %i
Requires=some-worker.service
Before=some-worker.service
BindsTo=some-worker.service

[Service]
PermissionsStartOnly=true
Type=idle
User=root
#EnvironmentFile=/etc/profile.d/optional_envvars.sh
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
TimeoutStartSec=10
RestartSec=10

[Install]
WantedBy=some-worker.service

/usr/bin/some-worker-start.sh:

#!/bin/bash
systemctl start some-worker@{1..10}

/etc/systemd/system/some-worker.service:

[Unit]
Description=manages some worker instances as a service, instance

[Service]
Type=oneshot
ExecStart=/usr/bin/sh /usr/bin/some-worker-start.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And now you can manage all instances with sudo systemctl some-worker (start|restart|stop)

Here is some boilerplate for your script.py:

#!/usr/bin/env python

import logging


def worker_loop():
    shutdown = False
    while True:

        try:
            if shutdown:
                break

            # Your execution logic here.
            # Common logic - i.e. consume from a queue, perform some work, ack message
            print("hello world")

        except (IOError, KeyboardInterrupt):
            shutdown = True
            logging.info("shutdown received - processing will halt when jobs complete")
        except Exception as e:
            logging.exception("unhandled exception on shutdown. {}".format(e))


if __name__ == '__main__':
    worker_loop()

GregL's answer helped me a great deal. Here is an example of a unit template I used in my code using the example above for a gearman job server. I made a shell script that lets me create X amount of "workers" using this one template.

[Unit]
Description=az gearman worker
After=gearman-job-server.service

[Service]
PIDFile=/var/run/gearman_worker_az%i.pid
Type=simple
User=www-data
WorkingDirectory=/var/www/mysite.com/jobs/
ExecStart=/usr/bin/php -f gearman_worker_az.php > /dev/null 2>&1
Restart=on-success
KillMode=process

[Install]
WantedBy=multi-user.target