journalctl stops working randomly after boot on DigitalOcean droplet

I am building servers from a custom image I made using Packer on DigitalOcean, and I'm having a recurring issue where journalctl doesn't have any logs:

# journalctl
No journal files were found.

Sometimes I have this issue with images I build, sometimes I don't. Sometimes one server built from the same image works and another doesn't.

I have also noticed that if I log in really quickly to the server, sometimes journalctl is initially working but breaks soon after (<1 minute later).

If I restart journald then it works as normal from then on, but I lose all the logs prior to that point.

It isn't random. The "randomness" is based on whether 10 minutes have elapsed between building the image, and starting the server – that is why sometimes the same image gives different results.

The issue is that you have not cleared the machine id in the image.

DigitalOcean inject a first-boot script via cloud-init to generate a new machine id if there was one in the base image. The heuristic this script uses to tell if a machine id is baked into the image is to check if the mtime on /etc/machine-id is more than 10 minutes ago.

/var/lib/cloud/scripts/per-instance/machine_id.sh:

# record timestamp on machine-id for testing
# If /etc/machine_id is over 10m old on first-boot, delete it
if [ -f /etc/machine-id ]; then
    if [ $DIFF -lt 600 ]; then
        exit 0
    fi
rm -rf /etc/machine-id
fi

The systemd journal stores log files on disk in a directory named after the machine-id, so changing the machine id makes it lose track of those logs.

You can see the logs from before the machine id changed by telling journalctl to load logs from the old directory, e.g:

journalctl -D /var/log/journal/4ff0f6fdda274a6b9d2b9287b8a15c81

The fix here is to clear the machine-id in your baked images. You can do this by removing /var/lib/dbus/machine-id and /etc/machine-id, or alternatively you can use the machine-id script from packer-virt-sysprep, which is a library of useful scripts for preparing a VM to be an image.