syslog-ng not flushing the pipe to external program

I have written a python script that takes log entries from syslog-ng and writes them to MongoDB (I couldn't do it with afmongodb driver because I need to do some special processing).

This is how it looks like in syslog-ng.conf:

destination d_mongodb_events {
  program("/home/test/syslog_piper.py"
    template("$UNIXTIME|$PRIORITY|$FACILITY|$SOURCEIP|$SEQNUM|$PID|$PROGRAM|$MSGONLY\n")
    flags(no_multi_line)
    flush_lines(1)
    flush_timeout(1000)
  );
};

And this is the script: (with logic removed)

import sys
try:
    lines = sys.stdin.readlines()
    for line in lines:
        # process `line` and save to DB

except Exception, e:
    f = open('/tmp/error.txt','ab')
    f.write(e)
    f.close()
    exit(0)

Script works - that is, if I run it from the command line it waits for input and inserts the lines in DB as soon as I press Ctrl+D - and exits.

With syslog-ng it is different. The entries are passed to the script, but only after I stop syslog-ng daemon. Also, while syslog-ng is running, I see my script running too (in the process list). I have tried setting flush_lines() and flush_timeout() (see configuration above) but I can't make syslog-ng flush its output.

My guess is that there is something wrong with the way I handle pipelines, but I can't figure it out. Can anyone spot the problem?

UPDATE: if I send 1000 messages some of them get pushed through, so I guess there is some buffering going on. Anyone knows which setting to tune?

SOLUTION: It looks like Python buffers a lot of its input/output. This is from the man pages:

 -u       Force  stdin,  stdout  and  stderr to be totally unbuffered.  On
          systems where it matters, also put stdin, stdout and  stderr  in
          binary  mode.   Note  that there is internal buffering in xread-
          lines(), readlines() and file-object  iterators  ("for  line  in
          sys.stdin")  which  is  not  influenced by this option.  To work
          around this, you will want to use "sys.stdin.readline()"  inside
          a "while 1:" loop.

So basically I had to change the program to use sys.stdin.readline(). Huge thanks to Janne.

Solution 1:

I have not hooked any Python scripts to syslog-ng, but with Perl scripts I have to turn off the output buffering before they work in real-time. In Perl-speak, that is $|=1.

I am not much a Python guy, but I guess running your Python script with -u or setting up variable PYTHONUNBUFFERED might help.

syslog-ng not flushing the pipe to external program

Solution 1:

Related

Recent Posts