Why do JBoss and Logrotate create log files full of NUL characters?

I've set up Logrotate to rotate my JBoss Application Server 4.2.2.GA logs nightly. After the log files have been rotated and JBoss starts writing to them again, the new log files begin with as many NUL characters as there were characters in the previous log file, followed by the new log messages. For example, if the JBoss server.log file was 5000 bytes long then, after rotation, the new server.log file will begin with 5000 NUL characters. After several days, server.log begins with NUL characters equivalent to the characters in all the previous days log files combined. It seems as if JBoss is remembering its position in the log file and picking up where it left off in the truncated file. Here is my logrotate config for JBoss:

/apps/jboss-4.2.2.GA/server/default/log/*log {
    daily
    rotate 30
    compress
    notifempty
    copytruncate
    missingok
    nocreate
}

I can't restart JBoss nightly because that would be too much downtime. I also can't use the log4j DailyRollingFileAppender as it doesn't delete old log files. Has anyone gotten logrotate working properly with JBoss?

We had the same problem for a file being written by log4j. The solution was to set the property "Append" for the FileAppender to "true". After this change, we have not seen this problem with files having NUL when rotated by an external program like logrotate.

From my experience - and the reason that we do not use logrotate with Log4j is that the way logrotate works is that it renames the files, then instructs the program to close its logs and reopen them with the old file name (which does not exist anymore), normally using the HUP signal.

But Log4j cannot be told to reopen its log files, so I see you use copytruncate to copy the files instead - the problem is that Log4j uses buffered writers that keep track of the current position of the file that is being written and when you truncate the log file log4j keeps writing from where it stopped writing before the truncate. Depending on your file system implementation this should create "files with holes" - i.e. the NULL characters that you see there do not really exist - the file is actually only as big as the actual data and the NULL character is the way your viewer represents the hole. Some file systems on the other hand do not support holes and will indeed fill the file with NULL characters when Log4j resumes writing.

I suggest - don't use logrotate, find some way to rotate the files within Log4j by using a RollingFileAppender (that does support removing old files) or using the DailyRollingFileAppender and a cronjob that removes old files externally (as it was meant to be done).