Why is syslog so much slower than file IO?

Solution 1:

The syslog calls both issue one send() to an AF_UNIX socket per call. Even if the syslogd discards the data it'll still have to read it first. All this takes time.

The writes to /dev/null also issue one write() per call but since the data is discarded it can be processed very quickly by the kernel.

The fprintf() calls only generate one write() for every 4096 bytes that are transferred, i.e. about one every eighty printf calls. Each only involves transferring data from libc's buffer to the kernel's buffers. The commit to the disk will be (in comparison at least) very slow, but in the absence of any explicit synchronization calls can happen later (perhaps even after the process terminates).

In short: syslog is slower than /dev/null because it's doing a lot of work and slower than printf to a file because of buffering.