How to generate a JSON log from nginx?

I'm trying to generate a JSON log from nginx.

I'm aware of solutions like this one but some of the fields I want to log include user generated input (like HTTP headers) which need to be escaped properly.

I'm aware of the nginx changelog entries from Oct 2011 and May 2008 that say:

*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
   access_log.
*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
   in an access_log.

but this still doesn't help since \xXX is invalid in a JSON string.

I've also looked at the HttpSetMiscModule module which has a set_quote_json_str directive, but this just seems to add \x22 around the strings which doesn't help.

Any idea for other solutions to log in JSON format from nginx?

Solution 1:

Finally it looks like we have good way to do this with vanilla nginx without any modules. Just define:

log_format json_combined escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

Note that escape=json was added in nginx 1.11.8. http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

Solution 2:

You can try to use that one https://github.com/jiaz/nginx-http-json-log - addition module for Nginx.

Solution 3:

You can try to use:

• addition module for Nginx nginx-http-json-log

• Use any language as done in nginx-json-logformat with example /etc/nginx/conf.d/json_log.conf

• A version of the Nginx HTTP stub status module that outputs in JSON format

PS: The if parameter (1.7.0) enables conditional logging. A request will not be logged if the condition evaluates to “0” or an empty string:

map $status $http_referer{
    ~\xXX  0;
    default 1;
}

access_log /path/to/access.log combined if=$http_referer;

It’s a good idea to use a tool such as https://github.com/zaach/jsonlint to check your JSON data. You can test the output of your new logging format and make sure it’s real-and-proper JSON.