Separate Nginx access log file for certain requests only

Solution 1:

cjc put me on the right track. Using access_log in an if statement by itself is not possible (You get a nginx: [emerg] "access_log" directive is not allowed here error). So the workaround is as follows:

if ($http_user_agent ~* (crawler) ) {
  set $crawler 'yes';
}
location ~ .* {
  if ($crawler = 'yes') {
    access_log /var/log/nginx/blockedbots.log;
    return 200;
    }
}

Solution 2:

access_log supports if:

(access_log path [format [buffer=size [flush=time]] [if=condition]];)

access_log /var/.... if $crawler;

Source:

http://nginx.org/en/docs/http/ngx_http_log_module.html

Solution 3:

You should be able to put an access_log directive inside an if block, according to the documentation:

http://wiki.nginx.org/HttpLogModule

So, you should be able to do something like:

if ($http_user_agent ~* (crawler) ) {
   access_log /path/to/other/log/file ;
}