Mystery stack traces in /var/log/messages

Solution 1:

It specifically looks as though ext4 is experiencing difficulty in writing data to the journal. My experience with ext3 was that the journal is used for writing the metadata you are about to commit to a safe place where it can at least be undone in the event of a system crash. Since writing to the journal is a non-critical, albeit important, function it probably just does an oops and rolls on. Since the system didn't crash at that moment, you only experience a delay during the time the ext4 kernel module was having issues with the journal.

So why is the filesystem having trouble with the journal? Look at the following questions to see if they spark any more ideas:

  • Did you put the journal on a separate device that is either having problems or whose %busy is pretty high?
  • If the disk is not local (iSCSI, ATAoE, or fiber channel), are you having communication problems, or can you prove that you aren't?
  • When was the last time you ran fsck to check for integrity errors?
  • If it a single, internal drive, have you tried to run the manufacturer's diagnostic tool to ensure that the disk is still performing to spec?
  • For internal drives, is S.M.A.R.T. enabled and are you using tools to watch the device?