imuxsock messages in syslog and system becomes unresponsive

Just seconds ago I resolved an issue where for almost an hour simple processes were hanging in busy cpu loop like in a lock or mutex. Syslog showed these messages:

Jan  9 12:15:43 host1 rsyslogd-2177: imuxsock lost 354 messages from pid 157711 due to rate-limiting

and the process 157711 was postfix. service postfix restart fixed the issue immediately.

It sounds like you may have an out-of-control process running on your system... Perhaps, even, your machine could have been compromised (?).

Those log messages are related to rsyslog's rate-limiting feature. I'm currently trying to pin down a similar problem on one of my machines, though mine seems a bit less severe -- I'm only losing about 100 log messages at a time. (E.g., I'm seeing "imuxsock lost 100 messages ...")

If anyone has any insights on tracking down the source of the "lost" logged messages, that could be most helpful. I believe my issue may be related to Postfix, but as the process IDs ("pid" numbers) given in the log message are no longer relevant (as, presumably, the processes no longer exists), it's a bit tricky to pin down...

My experience is that the message "rsyslogd-2177: imuxsock lost xxx messages from pid yyy due to rate-limiting" can appear in the logs several days after the log messages in question were dropped. This means that you can't find any other references to the given pid in the current log file, and so can't find any other messages from that pid that might tell you what it was.

Assuming you're using something like logrotate, grepping through the archived logs for the previous week will likely show the source of the dropped messages, and you'll see the messages that started the flood which might be enough to understand what was going on.