asynchronous logging via rsyslogd(8) and write buffer increase

On a pretty high traffic web site running in virtual containers (VMware) and lacking local storage, we've managed to increase throughput (requests per second) significantly by switching from logging directly into log-files (which reside on remote network storage) to rsyslogd.

Essentially we've switched from synchronous to asynchronous logging. The web server workers write using syslog(3) to some memory buffer and rsyslogd(8) sends the data to an actual file in parallel, and at its own pace, so processes don't block on IO when logging.

So far so good. The problem is that ocassionally rsyslogd is prevented from writing (e.g. a momentary/prolonged network outage) and the incoming buffer quickly fills up.

My questions are:

• Can a client ever block when writing to rsyslogd using syslog(3)?
• Is there a way to look at rsyslogd stats, e.g. how big/full is the buffer?
• Is there a way to increase the size of the rsyslogd incoming buffer?

Solution 1:

As far as I remember the default mode for the main message queue in rsyslog is fixed size array. It has a limit for 10k elements or so. Try to change this to linked list queue it should handle your occasional message bursts much better.

Yes, there are FixedArray and LinkedList queues.

Solution 2:

The answer to your first question is:

Yes, any call to syslog() is blocking. Maybe for a very short time, but it's still a synchronous call involving a file descriptor. See man 3 syslog for more detials.

Unless your servers use asynchronous architectures and primitives, there will always be some locking. Thsi can be mitigated, but not eliminated, for example by using a separatethread for logging. For the other two questions I don't really know but inspection to the rsyslogd source code (as well as to the one for the syslog() family of functions) is the only way to know.

More in general, if you move the logging to an external server via the UDP:514 "network syslog protocol", then you bring the possibilities to create locks to almost zero. With the drawback of possible loss of some logging during high loads.

First, in the "originating" servers you need to ensure all logging happens via syslog. For example, in Apache2 you need to specify:

ErrorLog "syslog:daemon"

For other servers please refer to the proper man page. If you cannot ensure this, please keep in mind that logging on filesystems can create

Second, in the originating rsyslogd configuration you ask to direct all syslog traffic for the facility you choose ("daemon" in this example) to one or more external syslog servers. In the rsyslog configuration file you can specify:

daemon.* @192.168.128.1
daemon.* @192.168.254.1

to have two copies of the logs to be sent to two different servers at the same time.

Third, in the destination server(s) you enable the reception of the syslog message over UDP:514. It's in the (destination) rsyslogd configuration file and is normally disabled by defualt (it'd be enough to remove the leading #s:

$ModLoad imudp
$UDPServerRun 514

Fourth, optional but highly recommended, I would also enable high resolution timestamps:

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Also this option is normally disabled by default (why on Earth?).