Missed kernel messages

On a CentOS 7.2 server I am trying to dump the kernel stack trace using echo t> /proc/sysrq-trigger. Some of the stack traces are not getting logged. I see the following error in /var/log/messages.

Jul 5 13:54:51 glusterfs1 journal: Missed 14295 kernel messages

It seems that some of the entries are getting lost. I need to find the applicable stack traces from this server. Please suggest what needs to be done to not miss the kernel messages.

Journald is logging almost everything and it copy the info to rsyslog via socket. This socket have a maximum throughput.

If you like it you can recompile it from source as "explained" here

https://cgit.freedesktop.org/systemd/systemd/commit/?id=5e8b767df6e18444d5aff2987b5e5603361ed528

But you can also limit the amount of data send trough the socket via the journald.conf in /etc/systemd

with the option

MaxLevelSyslog=

the default is debug, witch mean everything is send to rsyslog. But, historicaly, syslog are configurated to output only message with warning and greater importance. So you can make the configuration of journald to send only warning and greater importance log to syslog, this will do the trick on almost all setup. You will still be able to consult notice, info and debug log via the journalctl command, but until you have to dig in a problem those logs are'nt much relevant.

MaxLevelSyslog=warning

dont forget to restart your journald service afther any modification to journald.conf

systemctl restart systemd-journald

Another mentionned solution is to make rsyslog read the journald file instead of being flood by it via socket. But I did'nt find any usefull information on how to achieve this beheavior.

FYI : syslog level (also respected by journald)

0 emerg
1 alert
2 crit
3 err
4 warning
5 notice
6 info
7 debug