Walk me through the Linux log files (please)

I just tried loading a 2MB file in gedit and it silently died on me. I was wondering if anything might appear in a log file that might help me diagnose this: I checked syslog and found out it segfaulted. While doing this I realised that I don't really know anything about how logging is organised on *nix machines.

All I know at the mo is

1. Logs are typically stored in /var/log/... is there anywhere else that I should know about?

2. I'm familiar with application specific logs, such as apache's.

3. I understand that dmesg is the bootup log, and syslog is a general system log... is that right? Edit: Bobby says dmesg is also general purpose... what's the difference between the two?

So would someone mind taking me through the most useful logs? Are the two logs I mention in the final point the only general logs? And what are the funky numbers at the start of lines in dmesg? Seconds since startup?

Please include anything in your answers that you think would improve my understanding here and help me track down anomalies!

TIA

Andy

There is a starter at LinuxHomeNetworking: Quick HOWTO:Ch05:Troubleshooting Linux with syslog.

You might also like Ubuntu Tips: How To View System Log Files in GUI.

The Ubuntu LinuxLogFiles page is also a good reference
^{(posted in a comment by Pulse to another answer here).}

Many thanks to Pulse for recommending https://help.ubuntu.com/community/LinuxLogFiles. I've cut some bits out, and left out how to use syslogd and other essential commands, to leave this little guide for future reference. This is from an Ubuntu site and I don't know how much holds for other distros.

System Logs

System logs deal primarily with the functioning of the Ubuntu system, not necessarily with additional applications added by users. Examples include authorization mechanisms, system daemons, system messages, and the all-encompassing system log itself, syslog.

Authorization Log: /var/log/auth.log

The Authorization Log tracks usage of authorization systems, the mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd and so on.

Daemon Log: /var/log/daemon.log

The daemon log contains information about running system and application daemons such as the Gnome Display Manager daemon gdm, the Bluetooth HCI daemon hcid, or the MySQL database daemon mysqld.

Debug Log: /var/log/debug

The debug log provides detailed debug messages from the Ubuntu system and applications which log to syslogd at the DEBUG level.

Kernel Log: /var/log/kern.log

The kernel log provides a detailed log of messages from the Ubuntu Linux kernel. These messages may prove useful for trouble-shooting a new or custom-built kernel, for example.

Kernel Ring Buffer: dmesg

The kernel ring buffer is not really a log file per se, but rather an area in the running kernel you can query for kernel bootup messages via the dmesg utility. To see the messages, use this:

dmesg | less

By default, the system initialization script /etc/init.d/bootmisc.sh sends all bootup messages to the file /var/log/dmesg as well.

Messages Log: /var/log/messages

The messages log contains informational messages from applications, and system facilities. This log is useful for examining message output from applications, and system facilities which log to the syslog / sysklog daemon at the INFO level.

System Log: /var/log/syslog

The system log typically contains the greatest deal of information by default about your Ubuntu system. It may contain information other logs do not. Consult the System Log when you can't locate the desired log information in another log.

Application Logs

Many applications also create logs in /var/log. If you list the contents of your /var/log subdirectory, you will see familiar names, such as /var/log/apache2 representing the logs for the Apache 2 web server, or /var/log/samba, which contains the logs for the Samba server.

Apache HTTP Server Logs: /var/log/apache2

The default installation for Apache2 on Ubuntu creates a log subdirectory. Within this subdirectory are two log files with two distinct purposes:

• /var/log/apache2/access.log - records of every page served and every file loaded by the web server.
• /var/log/apache2/error.log - records of all error conditions reported by the HTTP server

CUPS Print System Logs: /var/log/cups/error_log

The Common Unix Printing System (CUPS) uses the default log file /var/log/cups/error_log to store informational and error messages.

Rootkit Hunter Log: /var/log/rkhunter.log

The Rootkit Hunter utility (rkhunter) checks your Ubuntu system for backdoors, sniffers and rootkits, which are all signs of compromise of your system.

Samba SMB Server Logs: /var/log/samba

The Server Message Block Protocol (SMB) server, Samba is popularly used for sharing files between your Ubuntu computer and other computers which support the SMB protocol. Samba keeps three distinct types of logs in the subdirectory:

• log.nmbd - messages related to Samba's NETBIOS over IP functionality (the network stuff)
• log.smbd - messages related to Samba's SMB/CIFS functionality (the file and print sharing stuff)
• log.[IP_ADDRESS] - messages related to requests for services from the IP address contained in the log file name.

X11 Server Log: /var/log/Xorg.0.log

The default X11 Windowing Server in use with Ubuntu is the Xorg X11 server. This log is helpful for diagnosing issues with your X11 environment.

Non-Human-Readable Logs

Some log files found in the /var/log subdirectory are designed to be readable by applications, not necessarily by humans. Some examples follow.

Login Failures Log: faillog

The login failures log located at /var/log/faillog is actually designed to be parsed and displayed by the faillog command.

Last Logins Log: lastlog

The last logins log at /var/log/lastlog should not typically be parsed and examined by humans, but rather should be used in conjunction with the lastlog command.

Login Records Log: who

The file /var/log/wtmp contains login records, but unlike /var/log/lastlog above, /var/log/wtmp is not used to show a list of recent logins, but is instead used by other utilities such as the who command to present a listed of currently logged in users.