systemd's journalctl: how to filter by message?

journalctl looks like a great tool for looking through logs, but I'm stuck on what feels like a simple ask: I want to see all cron messages that contain the phrase update-ipsets.

Of course I can do this

journalctl -u cron.service | grep update-ipsets

but then you lose all the other benefits of journalctl's output (colour coding, auto paging, live view etc.)

I've tried:

journalctl -u cron.service MESSAGE=update-ipsets
journalctl -u cron.service "MESSAGE=*update-ipsets*"
journalctl -u cron.service "MESSAGE=.*update-ipsets.*"
journalctl -u cron.service "MESSAGE=/.*update-ipsets.*/"

And you don't want to experiment by hitting tab after MESSAGE= - hangs the (zsh/Debian Jessie) shell and Ctrl-C didn't help either!

I sort of can't believe that it doesn't have this basic functionality built in, so I'm sure I must have missed something?

Thanks.

Solution 1:

Currently, journalctl does not support patterns or wildcards in field matches. grep is your best option.

I had the same problem, and I think that journalctl only searches for an exact match for VALUE when NAME=VALUE is passed as arguments.

My investigations:

1. man page

   From journalctl(1)

   The pattern is not mentioned in the description of the matches:

    [...] A match is in the format "FIELD=VALUE", e.g.
    "_SYSTEMD_UNIT=httpd.service", referring to the components
    of a structured journal entry. [...]
   

   The man page refers to a pattern when describing -u option only.

      -u, --unit=UNIT|PATTERN
          Show messages for the specified systemd unit UNIT 
          (such as a service unit), or for any of the units
          matched by PATTERN. 
   

2. Source code

   The function fnmatch in src/journal is used when searching for units only.

3. debug journalctl

   Enabling debug output you can see that the pattern is expanded only when using -u.

   $ SYSTEMD_LOG_LEVEL=debug journalctl -n1 -u gdm*
   ...
   Matched gdm.service with pattern _SYSTEMD_UNIT=gdm*
   Matched gdm.service with pattern UNIT=gdm*
   Journal filter: ((OBJECT_SYSTEMD_UNIT=gdm.service AND _UID=0) OR (UNIT=gdm.service AND _PID=1) OR (COREDUMP_UNIT=gdm.service AND _UID=0 AND MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1) OR _SYSTEMD_UNIT=gdm.service)
   ...
   

   All the matches are treated as exact, including UNIT:

   $ SYSTEMD_LOG_LEVEL=debug journalctl -n1 UNIT=gdm.*
   ...
   Journal filter: UNIT=gdm*
   ...
   

Solution 2:

Since systemctl --version version 237 there might be grep pattern support with -g/--grep switch, but it has to be compiled with PRCE2 support (it doesn't appear to be included in Debian Buster, >=242 is needed, it's possible to install in from buster-backports)

journalctl -g ipsets*

Without grep support you can still switch to cat output mode and use grep's matching:

journalctl -b -o cat --no-pager | grep "update-ipsets"

if you want a pager it's best to pipe the result to less. You can use invert matching -v / --invert-match to exclude certain messages

journalctl -b -o cat --no-pager | grep -v "ACPI" | less

Yet another option is to use json format:

journalctl -b -o json | jq -C . | less -R

which gives verbose output, single line

{
  "SYSLOG_IDENTIFIER": "kernel",
  "_MACHINE_ID": "d72735cff36a41f0a5326f0bb7eb1778",
  "_SOURCE_MONOTONIC_TIMESTAMP": "0",
  "__REALTIME_TIMESTAMP": "1614516018297106",
  "__CURSOR": "s=2b1deb3dba3e42e4a758cc8f011d19c5;i=1;b=c0f6b0e5143a4d3db9f04f00da1a4ff4;m=670dd9;t=5bc64cdc13912;x=ee44184076fc0590",
  "__MONOTONIC_TIMESTAMP": "6753753",
  "SYSLOG_FACILITY": "0",
  "_HOSTNAME": "w16",
  "PRIORITY": "5",
  "_BOOT_ID": "c0f6b0e5143a4d3db9f04f00da1a4ff4",
  "_TRANSPORT": "kernel",
  "MESSAGE": "Linux version 4.19.0-14-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)"
}

using jq you can easily filer messages:

$ journalctl -b -o json | jq '. | select(._COMM=="sensors")' | jq -r .MESSAGE | less
$ journalctl -b -o json | jq '. | select(._TRANSPORT=="kernel")' | jq -r .MESSAGE | head -n 1
Linux version 4.19.0-14-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)