How do view older journalctl logs (after a rotation maybe?)

I am running docker on ubuntu 16.04 and would like to view the logs. However, I am unable to view logs after what I am guessing is some sort of rotation or the logs grow to a certain size.

I have not made any changes to my journald.conf, so I am using defaults there. There are containers running so the docker log outputs quite a lot of data.

Examples of what I am seeing:

systemctl docker status confirms service has been active:

since Thu 2016-10-13 18:56:28 UTC

However, when I run something like:

journalctl -u docker.service --since "2016-10-13 22:00"

The only output I get is:

-- Logs begin at Fri 2016-10-14 01:18:49 UTC, end at Fri 2016-10-14 16:18:25 UTC. --

I can view the logs in that range as expected.

My question is: why can I not view older logs with journalctl, and how can I fix this issue so I can view the logs?

Solution 1:

It could be because you are trying to review the journal since the last boot, which seems likely to be the case inside a docker image.

On Ubuntu 16.04, the journal storage defaults to being in-memory. You can change the default to be persistent by opening /etc/systemd/journald.conf and changing the Storage= line from auto to persistent. You may need to restart journald by systemctl restart systemd-journald after the config file edit.

I think the journal should be persistent-by-default, so I opened a bug about that.

Solution 2:

The reason this happens is because of defaults on the size of journald files stored. There is more detail about this in the docs. It's worth reading the whole section I have linked to, but the defaults work like so:

journald will use 10% of the disk or 4G, whichever is smaller.

journald will leave free 15% of the disk or 4G, whichever is larger.

For viewing logs from the last boot, assuming you have Storage=persistent in your journald.conf, as the other answer notes, you can use the --boot=-1 flag on journalctl commands to get logs from just the previous boot.

In the case of the OP where they were sure the host had not been rebooted, the loss of logs was simply caused by the SystemMaxUse and/or SystemKeepFree defaults.

Note: I'm the OP and this question still has upvotes trickling in, so since I've gained more experience with journald (and rtfm) I am posting this here in the hopes it helps others.