Syslog-NG Error processing log message using IETF format

we've just changed our logging application on our AS/400 partitions and are now using a product called PowerTech SIEM agent for IBM I

The tool used to format messages using the old syslog convention and is apparently now capable of sending IETF messages (RFC 5424), however for some reason our Syslog-NG server is not able to process them, as if the format was not correct.

here is an example error we receive :

Feb  8 15:53:49 hostname syslog-ng[1489874]: Error processing log message: <38>1 2021-02-08T15:53:34.918160+1:00 HOSTNAME.fqn.here - - - - CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0

the event itself is :

<38>1 2021-02-08T15:53:34.918160+1:00 HOSTNAME.fqn.here - - - - CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0

Looking at the RFC, it looks like the contents are OK. they should respect :

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

in my case, i have :

<priority> is "38"
VERSION is "1"
ISOTIMESTAMP is "2021-02-08T15:53:34.918160+1:00"
HOSTNAME is "HOSTNAME.fqn.here"
APPLICATION is unknown (-)
PID  is unknown (-)
MESSAGEID  is unknown (-)
STRUCTURED-DATA is unknown (-)
MSG is "CEF:0!Powertech!SIEM Agent!4.2!AUDIT!TCD0003!6!src=0.0.0.0 dst=1.2.3.4 reason=Command executed msg=A command (WRKPTSA) was run by 12345/T555500/PC01AFB0"

we run syslog-NG 3.8

note that i did replace the hostname & IP on purpose, but kept the case.

i have tried multiple things in the syslog-ng configuration (in the source mainly), such as switching from the network() to the syslog() driver (also specified flags(syslog-protocol)), etc. but nothing changed), few examples of what i tried in the source :

source s_ISERIES        { syslog(transport("tcp") ip(0.0.0.0) port(51513)); };
source s_ISERIES        { network(transport("tcp") ip(0.0.0.0) port(51513) flags(syslog-protocol)); };

if i do not use syslog() or the syslog-protocol and use the basic network() driver without flags, the message gets "rewritten" (syslog-NG adds its own header and adds a timestamp and hostname at the beginning of the message, which i do not want).

is there a way to fix that ? do you spot any mistake in the event format ? and if not, is there a way to tell syslog-NG to process the "raw" message (take it and send it where i want without doing further modification of it).

thanks !

edit

after restarting and doing multiple checks, now we just don't have the error anymore. but logs aren't processed at all.

the whole "logging" configuration is quite simple, we've got :

the source defined like this :

source s_ISERIES        { syslog(transport("tcp") ip(0.0.0.0) port(51513)); }; 

the logging config defined like this :

log {
  source(s_ISERIES);
  destination(d_LOCALCOPY_ISERIES);
  destination(d_SPLUNK_ISERIES);
};

destinations defined like this :

destination d_LOCALCOPY_ISERIES   { file("/app/syslog-ng/logs/${YEAR}/${MONTH}/${DAY}/iseries/${HOST}.log" create-dirs(yes)); };

destination d_SPLUNK_ISERIES    { syslog("splunk-host.here.net" port(51513) send-time-zone("UTC")); }; 

and i can't see any message leaving the syslog-NG host when tcpdump'ing (i see them incoming, but not leaving the host). also, no local copy is created. so i "believe" syslog-NG is receiving the message but is unable to parse it for some reason. i tried the flags(no-parse) but it won't work either, as if it was not able to recognize the message at all on the "source" defined.

Solution 1:

As far as I can tell, IBM has never tested their RFC5424 exporter because their timestamp format is garbage and won't work with all too many exporters.

Per RFC5424 (emphasis mine):

TIMESTAMP = NILVALUE / FULL-DATE "T" FULL-TIME

FULL-DATE = DATE-FULLYEAR "-" DATE-MONTH "-" DATE-MDAY

DATE-FULLYEAR = 4DIGIT

DATE-MONTH = 2DIGIT ; 01-12

DATE-MDAY = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 based on month/year

FULL-TIME = PARTIAL-TIME TIME-OFFSET

PARTIAL-TIME = TIME-HOUR ":" TIME-MINUTE ":" TIME-SECOND [TIME-SECFRAC]

TIME-HOUR = 2DIGIT ; 00-23

TIME-MINUTE = 2DIGIT ; 00-59

TIME-SECOND = 2DIGIT ; 00-59

TIME-SECFRAC = "." 1*6DIGIT

TIME-OFFSET = "Z" / TIME-NUMOFFSET

TIME-NUMOFFSET = ("+" / "-") TIME-HOUR ":" TIME-MINUTE

Once you spot it it's actually quite clear in what you get since there's clearly a space included in your timestamp that shouldn't be:

ISOTIMESTAMP is "2021-02-08T15:53:34.918160+1:00 "