Rewriting logs received via syslog from another machine

I have Machine A, which is a pfsense installation, that sends logs via syslog to a Ubuntu box. The Ubuntu Box will have to rewrite the logs, to replace for instance host names and change format a bit.

The format is generally as follows

Mar  7 00:05:32 hostname service: field1 field2 field3 field4 field5 field6 field7

I would like the possibility of rewriting hostname, service and change order of fields, and filter out messages with a certain value in a certain field, as they are not interesting.

After filtering and handling, the messages should be written to disk in a log file, and sent to another machine via syslog.

Now, the logging part is trivial - simply set up rsyslogd to accept incoming messages, and forward those. However, I'm a bit stuck on the rewriting part. I'm not married to rsyslogd; any syslog-esque daemon will do.

Solution 1:

The question is a bit vague but I'll try to propose a possible solution. To rewrite messages rsyslog provides a number of modules one of which is mmfields. It splits an incoming message at a certain character (just one character) into fields and then allows to access these fields. For a message like

a=1 b=two c=3 d=four

the separator would be a blank and the fields are then accessible as $!f2, $!f3, $!f4, and $!f5. Unfortunately, the very first field ($!f1) is always empty because the message is preceeded by a space and that would be the first field. Thus, for the above message we get $!f1=="", $!f2=="a=1", $!f3=="b=two", $!f4=="c=3", and $!f5=="d=four".

rsyslog ships with other message modification modules as well but in lack of further details I chose this one. Store the following file as /etc/rsyslog.d/10-so.conf. Change the name according to the desired order of execution but keep the .conf extension.

# Load the "Message Modification" module "mmfields" to split
# incoming messages into fields at a certain separator:
module(load="mmfields")

# Format used for messages that are forwarded to another host.
# The fields are re-ordered and field #3 is omitted.
template(name="rewrite_forward" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag")
    constant(value=" ")
    property(name="$!f4")
    constant(value=" ")
    property(name="$!f2")
    constant(value=" ")
    property(name="$!f5")
}

# Format used for messages that are written to a local logfile.
# The format is almost the same as above, but lacks the priority,
# uses a different timestamp format, and ends with a "\n" as this
# is suitable for messages printed to a logfile.
template(name="rewrite_file" type="list") {
    property(name="timestamp")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag")
    constant(value=" ")
    property(name="$!f4")
    constant(value=" ")
    property(name="$!f2")
    constant(value=" ")
    property(name="$!f5")
    constant(value="\n")
}

if ( $programname == "so" ) then {

    # split message into fields at (exactly) one space character.
    # The "fields" can then be referred to as "$!f1", "$!f2", ..., "$!fN".
    # Note that "$!f1" will always be an empty string because the message
    # usually starts with a blank and that is considered to be the first
    # field. 
    action( type="mmfields" 
            separator=" " 
    )

    # If the second field is the string "b=2", then go ahead and log and
    # forward the message. Change the condition to your liking.
    if ($!f3 == "b=2") then {

        # write rewritten logmessage to local file
        action( type     = "omfile"
                file     = "/var/log/so-rewrite-fields.log"
                template = "rewrite_file"
        )

        # just for reference: write the unmodified message to the
        # very same logfile. Drop this for actual processing. It 
        # serves just as a debugging aid.
        action( type = "omfile"
                file = "/var/log/so-rewrite-fields.log"
        )

        # forward rewritten message to another host
        action( type = "omfwd"  
                target   = "127.0.0.1"  # change to actual destination
                port     = "514"        # change to actual destination
                protocol = "udp"        # change to actual destination
                template = "rewrite_forward"
        )

    }

    # no further processing:
    stop
}

Restart rsyslog (via sudo systemctl restart rsyslog.service) and try it out:

# A message that won't be logged because `$!f3 != "b=2"`:
logger -t so --id=$$ "a=1 b=3 txt=Hello number=$RANDOM"

# A message that will be logged because `$!f3 == "b=2"`:
logger -t so --id=$$ "a=1 b=2 txt=Hello number=$RANDOM"

The output of the second logger statement will be:

Mar 14 21:40:34 stratum9 so[3533]: txt=Hello a=1 number=6484        # rewritten msg
Mar 14 21:40:34 stratum9 so[3533]: a=1 b=2 txt=Hello number=6484    # original msg

To change the hostname, simply replace

constant(value=" ")
property(name="hostname")
constant(value=" ")

in the templates with

constant(value=" fromElsewhere ")

To change the syslogtag (what you called service), replace

    constant(value=" ")
    property(name="syslogtag")
    constant(value=" ")

with

constant(value=" otherService: ")

The output will then be:

Mar 14 22:05:51 fromElsewhere otherService: txt=Hello a=1 number=11763   # rewritten
Mar 14 22:05:51 stratum9 so[3533]: a=1 b=2 txt=Hello number=11763       # original

See here for further message properties.

Note that my approach (with mmfields) relies on the fields to always have the same order and does not easily allow for rewriting messages like a=1 b=2 to b=1 a=2 (reorder and change key-value-pairs). For that, another module might be more appropriate.

Solution 2:

As far as i know this could be achieved by using logstash elasticsearch and Kibana. I am trying to do the same and had been more or less successful by setting up an elk stack. Then using grok filters in logstash to break the syslog message into different fields and use those to match a pattern and send alerts out. Take a look at this guide it might give you some answers on where to begin.

This kind of setup already has built in filters for things like mysql logs or apache or nginx logs to begin with. Here is good overview of the elk stack features and architecture. I hope this helps.