Start service after AWS user-data has run

Use

[Unit]
…
After=cloud-final.service
…
[Install]
WantedBy=cloud-init.target

I ran into this as well. journalctl showed user data stuff running after multi-user and before the target cloud-init was reached.

<timestamp> <ip_addr> systemd[1]: Reached target Multi-User System.
[snip]
<timestamp> <ip_addr> systemd[1]: Starting Execute cloud user/final scripts...
<timestamp> <ip_addr> user-data[1592]: my user data stuff
[snip]
<timestamp> <ip_addr> systemd[1]: Started Execute cloud user/final scripts.
[snip]
<timestamp> <ip_addr> systemd[1]: Reached target Cloud-init target.

So I thought to have it be required/wanted by cloud-init instead of the usual multi-user target (which may be what op had), and it just worked.

The solution proposed by vlfig works. But, I think that it can be improved.

In my example, I'm trying to run the Puppet agent after cloud-init is finished.

In my case, we wanted Puppet to only run after Cloud Init is done with its set up. In order to achieve that, we need to modify the setup of its systemd unit.

The default execution order for the services and targets that interest us is the following: 1. cloud-init-local.service 2. cloud-init.service 3. cloud-config.target 4. basic.target 5. cloud-config.service 6. puppet.service 7. multi-user.target 8. cloud-final.service 9. cloud-init.target

(stag) dki@appbackend-i-0f0d3f6abd8520f12:~$ systemd-analyze --no-pager critical-chain puppet.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

puppet.service @17.737s
└─basic.target @17.291s
  └─paths.target @17.290s
    └─resolvconf-pull-resolved.path @17.290s
      └─sysinit.target @17.282s
        └─cloud-init.service @14.709s +2.572s
          └─systemd-networkd-wait-online.service @13.872s +832ms
            └─systemd-networkd.service @13.487s +377ms
              └─network-pre.target @13.482s
                └─cloud-init-local.service @6.983s +6.495s
                  └─var-lib.mount @11.888s
                    └─local-fs-pre.target @10.584s
                      └─keyboard-setup.service @6.506s +4.072s
                        └─systemd-journald.socket @6.430s
                          └─system.slice @6.416s
                            └─-.slice @3.772s

Dependency graph:

Systemd services dependency graph
Note: The green arrows mean After= and the gray arrows mean Wants=

By having puppet.service in /etc/systemd/system/multi-user.target.wants, systemd automatically creates a dependency order of the type After=puppet.service according to https://www.freedesktop.org/software/systemd/man/systemd.target.html#Default%20Dependencies

Dependency graph:

Puppet service dependency graph original
Note: The green arrows mean After= and the gray arrows mean Wants=

As a consequence, if we try to make it run After=cloud-init.target, it creates a dependency cycle and doesn't start at all.

[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found ordering cycle on puppet.service/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on cloud-init.target/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on cloud-final.service/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on multi-user.target/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Job puppet.service/start deleted to break ordering cycle starting with multi-user.target/start

In order to break this cycle, we need to disable the DefaultDependencies directive. By doing that, we are able to add the After=cloud-init.target to the Puppet unit in order to ensure that it waits for Cloud-Init to be done before it starts running. Both directives are done by creating the /etc/systemd/system/puppet.service.d/override.conf file, which overrides the package's original configurations without changing its original content. The advantage of it is that we can upgrade the package keeping the changes to only the directives we actually need.

Dependency graph:

Puppet service dependency graph afterwards
Note: The green arrows mean After= and the gray arrows mean Wants=

The resulting execution order is: 1. cloud-init-local.service 2. cloud-init.service 3. cloud-config.target 4. basic.target 5. cloud-config.service 6. multi-user.target 7. cloud-final.service 8. cloud-init.target 9. puppet.service

(sand) dki@supplier-integrations-i-035c65e75762aaabd:~$ systemd-analyze --no-pager critical-chain puppet.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

puppet.service @43.300s
└─cloud-init.target @43.298s
  └─cloud-final.service @33.272s +10.025s
    └─multi-user.target @33.246s
      └─splunk.service @13.815s +19.429s
        └─basic.target @13.463s
          └─sockets.target @13.463s
            └─docker.socket @13.453s +10ms
              └─sysinit.target @13.436s
                └─cloud-init.service @10.873s +2.557s
                  └─systemd-networkd-wait-online.service @9.182s +1.689s
                    └─systemd-networkd.service @9.034s +143ms
                      └─network-pre.target @9.032s
                        └─cloud-init-local.service @854ms +8.177s
                          └─var-lib.mount @7.587s
                            └─local-fs-pre.target @1.391s
                              └─keyboard-setup.service @569ms +822ms
                                └─systemd-journald.socket @564ms
                                  └─system.slice @564ms
                                    └─-.slice @423ms