Proper way to rotate Nginx logs

I would like to achieve rotation of nginx logs that:

1. would work without any extra software (i.e. - best if without "logrotate")
2. would create rotated files with names based on date

Best approach is something like PostgreSQL has - i.e. in its log_filename config variable I can specify strftime-style %Y-%m-%d, and it will automatically change log on date (or time) change.

Another approach from apache - sending logs via pipe to rotatelogs program.

As far as I was able to search - no such approach exists. All I can do, is to use logrotate with dateext option, but it has its own set of drawbacks, and I'd rather use something that works like |rotatelogs or log_filename in PostgreSQL.

While the world is divided on whether the humble named pipe is friend or foe, it is probably the simplest solution to your problem. It does have a few drawbacks (in that you need to create the pipes ahead of time), but it eliminates the need for a cron and allows you to used the logging pipe-filter of your choice.

Here's an example using cronolog on access.log:

1. Pick a path for our named pipe. I intend to keep my logs in /var/log/nginx, so I'll put my pipes there as well. The name is up to you; I append .fifo, and it's access.log, so mine will be at /var/log/nginx/access.log.fifo.
2. Delete the file if it exists.

3. Make a named pipe for the logfile:

   mkfifo /var/log/nginx/access.log.fifo
   

4. Configure nginx.conf to point the log at the pipe you just made:

   access_log /var/log/nginx/access.log.fifo;
   

5. Modify your init.d script to start the log rotator listening to the pipe before we start the server:

   LOGS="/var/log/nginx"
   pkill -f "/usr/sbin/cronolog --symlink $LOGS/access.log"
   ( cat $LOGS/access.log.fifo | /usr/sbin/cronolog --symlink $LOGS/access.log "$LOGS/%Y/%m/%d/access.log" ) &
   

   A similar commandline would be used for rotatelogs should you prefer it to cronolog -- see their docs for the syntax.

   If your distrobution has a start-stop-daemon, you should be using that instead, as it theoretically has whatever special knoweldge about your platform there is, and taking care of pkill for you. Simply wrap the command in a script, and pass it as --exec to start-stop-daemon in your init.d/nginx.

I've written a simple program, datelog, to split common logs based on the logged date, as opposed to current system time when the log line is seen by the program. This may or may not be exactly what cronolog or another log splitter does already but it was quicker to write my own than to find out what others do.

Using the year and month in the logged request, the line is then written to a file or pipe which includes the YYYYMM computed from the logged data. Yes this is somewhat specific for the common log format. The first [ is assumed to delimit the date. Beware of IPv6 addresses. :)

For log analysis it is important that each log really only contains the requests for each respective month, and each log should ideally be complete for correct analysis results. It's not enough to determine filename based on current time within the log splitter, because a slow request starting at 23:59:59 will then end up in the log file for the wrong month.

I use this with nginx by way of a named fifo which is checked to exist before nginx is started. Note that there is a tradeoff in the program between error detection and buffered output, where datelog currently prefers buffered output for performance reasons, so please make sure that your setup really works, especially when using shell pipes, in order to not lose any log data.

Source code: http://stuge.se/datelog.c

Please feel free to send me any feedback and of course patches!