Is it good practice to use NTFS Compression on IIS Log folders?
Is it a good practice to use NTFS Compression on a IIS Log folders and files?
I was able to go down from 20GB to 7GB by doing this. The IIS logs are per day, and have an average size of 20MB but some extreme days have 200MB.
I'm wondering if IIS has to open the whole file in memory, forcing NTFS to unzip 20MB (or 200MB in extreme case) each time? Or is there some magic that allows IIS to append content? What's the system impact? Could it become a problem if we grow our traffic?
Should I split them per hour instead of per day?
Any official Microsoft paper on this? I could not locate one.
As Evan already gave a general answer, I like to address two of your sub-questions:
Does IIS flushes logs every X minutes?
http.sys, the kernel mode part of IIS is responsible for logging and it buffers the data in memory before writing it to the log files. I'm not certain but I don't think it does the flushing every x seconds, more likely after its buffer is getting full.
Does the whole file need to be read when adding a single line?
No, NTFS writes updates to a file into its own cache and then compresses and appends the data asynchronously to the file. Writing to a compressed file is not significantly slower than to an uncompressed file.
So there should be no problem with using NTFS compression on IIS log files.
Sources:
IIS 7 Resource Kit, Chapter 15:Logging - Microsoft Press 2008
Windows Internals 6th Edition Part2, Chapter 12: File Systems Microsoft Press 2012
I compress my IIS logs on a lot of IIS servers, albeit mainly servers that are hosting Outlook Web Access/App or low volume web sites. I have no problems doing it, and quite like the disk space savings.
In general, you're trading CPU for storage by making this decision. If you're CPU-bound to begin with then this probably isn't a good tradeoff. For my OWA servers, which can grow gigabytes of logs a day (thanks ActiveSync devices) I think the tradeoff is a good one.
The NTFS filesystem driver handles the compression, so it doesn't change how IIS writes to the files.
Edit:
You are, potentially, also trading-off some I/O bandwidth and IOPS, too. If you're a high enough volume that your log writes are a significant consumption of I/O resources you could see a decline in I/O consumption from enabling compression, too.
The only way you're going to tell how this impacts you is to benchmark it yourself. Take a baseline with compression disabled and then enabled and compare them. There's no magic wand to wave to know how it will impact you-- there's just too many non-deterministic factors in play.