Server locking up, /var/log/messages reports "backlog limit exceeded"

We have a CentOS OS that became unresponsive this morning to external network traffic. It is a virtual machine. I was able to reboot the VM. After logging back in, I found the following in the /var/log/messages file, repeating over and over, up to the point of the reboot:

Jan 21 06:53:01 PBX kernel: audit: backlog limit exceeded
Jan 21 06:53:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: printk: 8 messages suppressed.
Jan 21 06:54:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: audit: audit_lost=1130 audit_rate_limit=0 audit_backlog_limit=320

I read on another forum that the following command could identify the source of the backlog traffic:

[root@PBX log]# aureport --start today --event --summary -i

Event Summary Report
======================
total  type
======================
486  USER_ACCT
486  CRED_ACQ
486  USER_START
485  LOGIN
477  CRED_DISP
477  USER_END
6  USER_LOGIN
3  USER_AUTH
2  CONFIG_CHANGE
2  CRED_REFR
1  DAEMON_START

Can anyone advise me as to what next steps I should take in preventing this issue from happening again? I'm not particularly familiar with the purpose of the backlog or what the output of the event summary report means.


You can increase the backlog by modifying -b 320 in /etc/audit/audit.rules to something larger and see if it has any effect, but these amounts you show us still very few audit results, so I doubt the audit error has anything much to do with the system freezing in itself. Its probably just a sympthom of something else happening.

Check /var/log/audit/audit.log to see what events have been logged to see if they can be of any use to your debugging.


There is multiple solution:

  1. To lengthen the backlog, add or edit /etc/audit/audit.rules by adding or editing "-b 320" to "-b 8192".
  2. change the priority by editing priority_boost from 3 to 4 or 5 in /etc/audit/auditd.conf.

To find out about what problem cause this issue, run aureport --start today or aureport --start today --event --summary -i