How long do you keep log files?
I have an application which writes its log files in a special folder. Now I'd like to add a functionality to delete these logs after a defined period of time automatically. But how long should I keep the log files? What are "good" default values (7 or 180 days)? Or do you prefer other criteria (e.g. max. used disk space)?
Solution 1:
There's no single answer. Factors to consider:
- legal requirements for retention or destruction
- company policies for retention and destruction
- how long the logs are useful
- what questions you're hoping to answer from the logs
- how much space they take up
I've got some logs that roll over in days, and others that are kept for years.
I tend to keep access logs longer (in some cases, forever, once I've stripped identifying information out of them) than error logs as in theory, I don't need them once I've fixed the problems.
I keep some other debugging logs (showing user activity, how they're interacting with the system, etc.) around so we can see how things change with each version.
...
And this reminds me of anecdote when I worked for a university -- management brought in an outside consultant to do a third-party review of the webserver we were building. I was told to hand over 3 months of logs for them to review if we had sized the system appropriately. I knew that was a sign they didn't know what the hell they were doing, as universities are cyclic -- webserver load trended up over time, but with spikes at the beginning of each year, finals time, etc.
Solution 2:
Depends on the requirements... there can be legal requirements for record keeping, and also you have to factor in how long they will be helpful for troubleshooting.
Solution 3:
In addition to legal requirements and utility considerations, it depends quite a bit on how much disk space your logs are chewing up. I've seen very verbose programs rotate their logs every day, while very quiet programs rotate almost never.
Depending on your system, it might be worth considering the "logrotate" package instead of handling log rotation internally. Just place a file with reasonable defaults in /etc/logrotate.d/ and let your users modify it if they deem necessary.