Best way to archive journald logs in a space-efficient way?

Solution 1:

journald rotates its own files, so it's safe to copy the old files (which always have an @ in the filename) elsewhere, compress them, or do whatever you want with them. If it's necessary to look into them, journalctl can be pointed at specific journal files with the --file command line option.

You should also see journald.conf for options to limit journald's disk usage.

Solution 2:

Well it's true that xz compression might give you the best compression level.

Yet xz compressed logs can not be queried with journalctl.

The other approach is to use filesystem compression. I use btrfs for this with zstd compression and it gives me about 10x space usage reduction and logs are still available for interaction via journalctl. I do not recommend using btrfs volume for /var/log/journal as journalctl keeps marking its directories with No_COW attribute (see lsattr) which prevents compression. It's possible to overcome this like I've done in my setup, but it is more fragile and complicated. Another way is to move old log files to separate compressed dir. Something like this:

mkfs.btrfs "$BLOCK_DEVICE_PATH"
mkdir /var/log/journal_archive

# With my data zstd compression level gives best compression/speed ratio.
# Your mileage may vary.
#Default level 3 is quite good so you might just stick with the default
mount -o compress=zstd:8 "$BLOCK_DEVICE_PATH" /var/log/journal

# To archive old logs (older than a week):
find /var/log/journal/$MACHINEID/ -name '*@*.journal' -mtime +7 -exec mv -v {} /var/log/journal_archive \;

# To query old logs
journalctl --directory=/var/log/journal_archive

# Cleanup archive to keep 20GB of uncompressed files
journalctl --vacuum-size=20G --directory /var/log/journal_archive