Sending logs to Graylog2 server

This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):

In Apache, I define a CustomLog format called graylog2_access which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.

Here is the custom format that it creates (human readable):

{ 
 "version": "1.1",
 "host": "%V",
 "short_message": "%r",
 "timestamp": %{%s}t,
 "level": 6,
 "_user_agent": "%{User-Agent}i",
 "_source_ip": "%a",
 "_duration_usec": %D,
 "_duration_sec": %T,
 "_request_size_byte": %O,
 "_http_status": %s,
 "_http_request_path": "%U",
 "_http_request": "%U%q",
 "_http_method": "%m",
 "_http_referer": "%{Referer}i"
}

For the Apache config, here is a copy/paste version:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

Then in your host configuration:

CustomLog "|nc -u graylogserver 12201" graylog2_access

You can also send your log files to graylog2 server using this simple command:

tail -F -q $yourlogfile |   while read -r line ; do   echo "<7> $hostnamesendingthelog $line" | nc -w 1 -u $graylogserver 514;   done;

I use this mainly for test purposes to determine if my log format is adapted for easy querying in graylog2. For production use you will wan't to set up rsyslog or syslog-ng.

You can probably tail your rails log file and see what happens.

Graylog2 only accepts logs in two formats: standard syslog and the Graylog extended log format (aka GELF). Arbitrary logs on disk are going to need some third-party process to consume the logs and translate it into a form that's useful for you.

Take a look at Logstash. Most people think of it as a tool to index logfiles using ElasticSearch, but it also contains a general purpose "log router" that lets you tail a bunch of files on disk and send them to a logging component like Graylog.

You can use the apache2gelf scripts from here.