An alternative to /usr/bin/logger for getting logs from Apache to syslog-ng

We currently use logger to get our Apache access logs to syslog-ng with a line like this one in each vhost:

CustomLog "|/usr/bin/logger -p local1.info  -t www_main" combined

It seems that almost every tutorial or advice I can find regarding Apache and syslog-ng uses this method to get the logs into syslog-ng. (Example). The rest use named pipes and the pipe() source. (Example).

The trouble with this is that logger splits any lines longer than 1024 bytes and sends them as separate log entries, meaning that some log entries end up on separate lines in the final destination log file on the logging box.

Using a named pipe and the pipe() source in syslog-ng solves the split lines problem but comes with its own set of minor problems and annoyances. To name a few, the named pipe has to be created before both Apache and syslog-ng are started, syslog-ng must be started before Apache is started and tagging the logs (which is done above with -t) must now be done in the syslog-ng configuration file rather than in the vhost.

This page suggests writing a short Perl script to use in place of /usr/bin/logger.

I was wondering if anyone knows of any alternatives to logger, preferably written in a natively compiled language or maybe an updated version of logger that doesn't split long lines in half.

I would also be interested in hearing of any other solutions to the problem of long log lines being split, such as a way of combining them back together using syslog-ng once they reach the logs servers.

Solution 1:

Updating my answer after looking into this more.

This seems to a limit with /usr/bin/logger, which is expected to conform with the syslog RFCs. http://www.faqs.org/rfcs/rfc3164.html says:

The total length of the packet MUST be 1024 bytes or less.

If you to send more then 1024 characters to syslog via the commandline (outside of Apache), you'll run into this same limit.

Keep in mind that the 1024 character limit probably exists elsewhere. I think the maximum size for a HTTP GET is 1024 characters, and I seem to recall that some printf library routines have a hard limit of 1024 characters (There was a security alert a couple years ago regarding the 1024 character limit regarding some syslog/ string printing utilities, if I remember right). So, it seems that your options are:

3) Try to stop your HTTP applications from writting long log messages. This is easier said then done. 1) Recompile logger and increase this limit. If you do this, keep in mind that you're changing a core utility and this may result in unexpected behavior. To mitigate this, put this utility in /usr/local/bin or /opt/bin. Do not replace /usr/bin/logger. 2) Don't send from Apache to syslog. Something like the following should work around the 1024-character limit, since this doesn't use syslog.

CustomLog logs/access_log

4) http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html uses sys::syslog and seems to be a reasonable alternative to /usr/bin/logger. You need to check sys::syslog for this same 1024 character limit. It's Perl, and should be easy to override.

Old answer:

It looks like this limit is tunable within syslog-ng, according to http://www.campin.net/syslog-ng/faq.html

syslog defaults to 1024 byte long messages, but this value is tunable in syslog-ng 1.5 where you can set it to a higher value.

options { log_msg_size(8192); };

Solution 2:

Recent versions of util-linux provide logger capable of taking --size parameter:

--size size
    Sets the maximum permitted message size to size.
    The default is 1KiB characters, which is the limit traditionally used
    and specified in RFC 3164. With RFC 5424, this limit has become flexible.
    A good assumption is that RFC 5424 receivers can at least process 4KiB messages.

    Most receivers accept messages larger than 1KiB over any type of syslog protocol.
    As such, the --size option affects logger in all cases (not only when --rfc5424
    was used).

The RFC 5424 protocol has been the default for logger since version 2.26.