Why does dd sometimes not wait until data is written?

Solution 1:

Use this instead:

sudo dd if=install57.fs of=/dev/sdc conv=fsync

This calls the fsync() after every write() system call. This forces dd not to cache anything. See this part of the manpage of fsync (man 2 fsync):

fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache 
pages for) the file referred to by the file descriptor fd to the disk device (or other 
permanent storage device) where that file resides. The call blocks until the device reports 
that the transfer has completed. It also flushes metadata information associated with the 
file (see stat(2)).

This is the kernels default behavior. Linux kernels manage write and read caches like this: When the write() syscall is issued, the data is quickly written to the cache and a write-completed status is sent to the process. When the buffer is needed or when there is free time on the bus, the data is written from the cache to the hard disk.

Solution 2:

This happens because Linux, and most other operating systems, cache both reads and writes. In most cases, this makes your operating system more responsive.

If you want to ensure the cached data has been written, you want to use sync, as you know. Linux exposes a significant number of settings you can tune as well. This article gives a good overview of some settings. You could set the vm.dirty_background_bytes to 0, for example, to ensure the kernel kicks off the flusher threads immediately.

Solution 3:

sync(8) - Linux man page :

The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance, but if the computer crashes, data may be lost or the file system corrupted as a result. sync ensures that everything in memory is written to disk.

Note: unmount (or eject) automatically calls sync which "hides" this in normal filesystem usage.