Server locking up, /var/log/messages reports "backlog limit exceeded"
We have a CentOS OS that became unresponsive this morning to external network traffic. It is a virtual machine. I was able to reboot the VM. After logging back in, I found the following in the /var/log/messages file, repeating over and over, up to the point of the reboot:
Jan 21 06:53:01 PBX kernel: audit: backlog limit exceeded
Jan 21 06:53:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: printk: 8 messages suppressed.
Jan 21 06:54:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: audit: audit_lost=1130 audit_rate_limit=0 audit_backlog_limit=320
I read on another forum that the following command could identify the source of the backlog traffic:
[root@PBX log]# aureport --start today --event --summary -i
Event Summary Report
======================
total type
======================
486 USER_ACCT
486 CRED_ACQ
486 USER_START
485 LOGIN
477 CRED_DISP
477 USER_END
6 USER_LOGIN
3 USER_AUTH
2 CONFIG_CHANGE
2 CRED_REFR
1 DAEMON_START
Can anyone advise me as to what next steps I should take in preventing this issue from happening again? I'm not particularly familiar with the purpose of the backlog or what the output of the event summary report means.
You can increase the backlog by modifying -b 320
in /etc/audit/audit.rules
to something larger and see if it has any effect, but these amounts you show us still very few audit results, so I doubt the audit error has anything much to do with the system freezing in itself. Its probably just a sympthom of something else happening.
Check /var/log/audit/audit.log
to see what events have been logged to see if they can be of any use to your debugging.
There is multiple solution:
- To lengthen the backlog, add or edit
/etc/audit/audit.rules
by adding or editing "-b 320" to "-b 8192". - change the priority by editing priority_boost from 3 to 4 or 5 in
/etc/audit/auditd.conf
.
To find out about what problem cause this issue, run
aureport --start today
or
aureport --start today --event --summary -i