Logging not working on Centos 7

Long story short - somehow I managed to screw up rsyslogd service on Centos 7 server. Now only thing that appears in /var/log/messages is:

Dec 14 22:13:27 {myhostname} rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1286" x-info="http://www.rsyslog.com"] exiting on signal 15.

Dec 14 22:13:27 {myhostname} rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1477" x-info="http://www.rsyslog.com"] start

Other logs (secure, cron, etc.) are empty too. Restarting service and server didn't help. Messages sent with logger doesn't end up in /var/log/messages file. rsyslogd service is running. Configuration file seems to be correct. I can see all events in journalctl logs. I already searched through several posts similar to my problem, but none of them had the solution.

While fastest solution might be reinstalling server, I would like to figure out what caused this problem and how to solve it, in case it happens in the future. Server is running simple LEMP stack, and last major change to system was hostname change.

Solution 1:

This was the solution, not permanent, though:

rm -f /var/lib/rsyslog/imjournal.state

service rsyslog restart

Source: https://bugzilla.redhat.com/show_bug.cgi?id=1088021

Solution 2:

You may try to debug rsyslog to find out the problem causing it's malfunctioning.

Getting debug information from a running Instance

It is possible to obtain debugging information from a running instance, but this requires some setup. We assume that the instance runs in the background, so debug output to stdout is not desired. As such, all debug information needs to go into a log file.

To create this setup, you need to point the RSYSLOG_DEBUGLOG environment variable to a file that is accessible during the while runtime (we strongly suggest a file in the local file system!) set RSYSLOG_DEBUG at least to DebugOnDeman NoStdOut make sure these environment variables are set in the correct (distro- specifc) startup script if you do not run rsyslogd interactively

These settings enable the capability to react to SIGUSR1. The signal will toggle debug status when received. So send it one to turn debug loggin on, and send it again to turn debug logging off again. The third time it will be turned on again ... and so on.

On a typical system, you can signal rsyslogd as follows:

kill -USR1 `cat /var/run/rsyslogd.pid`

Important: there are backticks around the “cat”-command. If you use the regular quote it won’t work. The debug log will show whether debug logging has been turned on or off. There is no other indication of the status.

Step by step how-to here: How to use debug on demand?

Solution 3:

This was my solution - not very nice an of course you loose all past journal:

rm -rf /var/log/journal/*
reboot

I am still not sure why it all started because all I did was a reboot after about 6 months of uptime.

Solution 4:

My rsyslogd.service refuses to start showing

Failed to start System Logging Service.
Unit rsyslog.service entered failed state.
rsyslog.service failed.
rsyslog.service holdoff time over, scheduling restart.
start request repeated too quickly for rsyslog.service
Failed to start System Logging Service.
Unit rsyslog.service entered failed state.
rsyslog.service failed.

I killed the service to be sure it's not running

killall rsyslogd

then removed the pid file

rm /var/run/syslogd.pid

and started the service again

systemctl start rsyslog.service

now the status is ok

# systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-02-06 11:32:33 CET; 2s ago

and the /var/log/*log files are growing again.