rsyslogd not monitoring all files
So.. I've installed Logstash, and instead of using the logstash shipper (because it needs the JVM and is generally massive), I'm using rsyslogd with the following configuration.
# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local6.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
In /etc/rsyslog.d/logstash.conf there are 28 file monitor blocks using imfile
$ModLoad imfile # Load the imfile input module
$ModLoad imklog # for reading kernel log messages
$ModLoad imuxsock # for reading local syslog messages
$InputFileName /var/log/rabbitmq/startup_err
$InputFileTag rmq-err:
$InputFileStateFile state-rmq-err
$InputFileFacility local6
$InputRunFileMonitor
....
$InputFileName /var/log/some.other.custom.log
$InputFileTag cust-log:
$InputFileStateFile state-cust-log
$InputFileFacility local6
$InputRunFileMonitor
....
*.* @@10.90.0.110:5514
There are 28 InputFileMonitor blocks, each monitoring a different custom application logfile..
If I run
[root@secret-gm02 ~]# lsof|grep rsyslog
rsyslogd 5380 root cwd DIR 253,0 4096 2 /
rsyslogd 5380 root rtd DIR 253,0 4096 2 /
rsyslogd 5380 root txt REG 253,0 278976 1015955 /sbin/rsyslogd
rsyslogd 5380 root mem REG 253,0 58400 1868123 /lib64/libgcc_s-4.1.2-20080825.so.1
rsyslogd 5380 root mem REG 253,0 144776 1867778 /lib64/ld-2.5.so
rsyslogd 5380 root mem REG 253,0 1718232 1867780 /lib64/libc-2.5.so
rsyslogd 5380 root mem REG 253,0 23360 1867787 /lib64/libdl-2.5.so
rsyslogd 5380 root mem REG 253,0 145872 1867797 /lib64/libpthread-2.5.so
rsyslogd 5380 root mem REG 253,0 85544 1867815 /lib64/libz.so.1.2.3
rsyslogd 5380 root mem REG 253,0 53448 1867801 /lib64/librt-2.5.so
rsyslogd 5380 root mem REG 253,0 92816 1868016 /lib64/libresolv-2.5.so
rsyslogd 5380 root mem REG 253,0 20384 1867990 /lib64/rsyslog/lmnsd_ptcp.so
rsyslogd 5380 root mem REG 253,0 53880 1867802 /lib64/libnss_files-2.5.so
rsyslogd 5380 root mem REG 253,0 23736 1867800 /lib64/libnss_dns-2.5.so
rsyslogd 5380 root mem REG 253,0 20768 1867988 /lib64/rsyslog/lmnet.so
rsyslogd 5380 root mem REG 253,0 11488 1867982 /lib64/rsyslog/imfile.so
rsyslogd 5380 root mem REG 253,0 24040 1867983 /lib64/rsyslog/imklog.so
rsyslogd 5380 root mem REG 253,0 11536 1867987 /lib64/rsyslog/imuxsock.so
rsyslogd 5380 root mem REG 253,0 13152 1867989 /lib64/rsyslog/lmnetstrms.so
rsyslogd 5380 root mem REG 253,0 8400 1867992 /lib64/rsyslog/lmtcpclt.so
rsyslogd 5380 root 0r REG 0,3 0 4026531848 /proc/kmsg
rsyslogd 5380 root 1u IPv4 1200589517 0t0 TCP 10.10.10.90 t:40629->10.10.10.90:5514 (ESTABLISHED)
rsyslogd 5380 root 2u IPv4 1200589527 0t0 UDP *:45801
rsyslogd 5380 root 3w REG 253,3 17999744 2621483 /var/log/messages
rsyslogd 5380 root 4w REG 253,3 13383 2621484 /var/log/secure
rsyslogd 5380 root 5w REG 253,3 7180 2621493 /var/log/maillog
rsyslogd 5380 root 6w REG 253,3 43321 2621529 /var/log/cron
rsyslogd 5380 root 7w REG 253,3 0 2621494 /var/log/spooler
rsyslogd 5380 root 8w REG 253,3 0 2621495 /var/log/boot.log
rsyslogd 5380 root 9r REG 253,3 1064271998 2621464 /var/log/custom-application.monolog.log
rsyslogd 5380 root 10u unix 0xffff81081fad2e40 0t0 1200589511 /dev/log
You can see that there are nowhere near 28 logfiles actually being read.
I really had to get one file monitored, so I moved it to the top, and it picked it up, but I'd like to be able to monitor all 28+ files, and not have to worry.
OS is
Centos 5.5
Kernel 2.6.18-308.el5
rsyslogd 3.22.1, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: Yes
FEATURE_NETZIP (message compression): Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
Questions:
Why is rsyslogd only monitoring a very small subset of the files? How can I fix this so that all the files are monitored?
Solution 1:
I know some of these are obvious, but here come the things I would try...
- Verify that the state file names are unique
- Verify that every
$InputFileNamepoints to an existing regular file - Give it some time. The default polling interval is 10 seconds which could feel like an eternity
-
Clearly demonstrate what is not working. Your question says that the files are not being monitored based solely on the output of
lsof. You don't mention if you actually tried writing to one of the problematic files, waited 10 seconds, and nothing showed up in the output log. And also demonstrate how you verify the files that are being monitored correctly. - Try to take out some of the files being monitored. Maybe it is a problem with only one of the monitored files which makes rsyslog ignore the rest of the files, too. It would help to know which one it is.
And if it helps, I have a CentOS 5.5 with the same version of rsyslog. And I tried it out with 40 files and rsyslog was able to monitor all of them. So I know that it can work correctly.
Solution 2:
The question already has an accepted answer, but in case someone else needs this.
According to the rsyslog documentation the default value for PersistStateInterval is 0 ($InputFilePersistStateInterval in the context of the imfile module) meaning that the rsyslog file is only written to when the inputfile is closed. Meaning that it will appear to not work if you have a continuously open log file.
If you run into this problem, try setting $InputFilePersistStateInterval to a low value (for testing only), say 2 or 3, then once the input file has that number of lines written to it, they will be passed on to rsyslog.