Make consistent copy of maildir

I just read the documentation for Maildir in Dovecot and few other documents about Maildir and Maildir++. I hope that I did not miss anything important.

Maildir is designed to work without locks. Most required operations are atomic on modern file systems. This means that you do not have to care much about race conditions like inconsistent read. But there are still some issues if you want to backup a Maildir while the mail-server is running.

Issues

• Backing up and restoring tmp/ is useless. Each Maildir contains the directories new/, cur/ and tmp/. Directory tmp/ contains mails which are being written to disk at the moment. They are moved to new/ when they are written successfully. This means that files in tmp/ may not be complete yet. Even if the file is complete, the process which was writing the file is not running anymore after restoring the backup. This means if such file is restored, it will never be added to the mailbox and it may never be removed.

• It may be sensible to exclude dovecot-uidlist.lock from backup. Dovecot uses an extension of Maildir called Maildir++. This extension need locking. Acquiring the lock for reading is not required by the extension but it may be sensible to exclude the lock-file from backup.

• Acquire dovecot-uidlist.lock or use snapshot of filesystem. Basically, you can just copy the directory but it is possible to miss some mails by race conditions. The reason is that listing and copying the content of a directory (recursively) is not atomic. This means that it is possible to miss an email while creating the backup when a user is changing a tag (e.g. seen/unseen) or moving the mail. To handle such situations, Dovecot acquires the lock-file (same as above) for every action. If you are using Dovecot, you can solve the problem by acquiring the lock before creating the backup. This can be done with /usr/lib/dovecot/maildirlock. Another possibility is to use snapshots of your filesystem. Since creating a snapshot is atomic, the problem does not occur on snapshots.

As a quick summary: If you want to make a backup while Dovecot is running, you should first acquire the lock-file dovecot-uidlist.lock or create a snapshot of you filesystem. Then, you can copy the Maildir. It may be sensible to exclude tmp/ and dovecot-uidlist.lock from your backup.

An updated answer to this question is to use dovecot's doveadm or dsync command. example:

dsync -f -u <user> backup maildir:<backup_location>

There is a dovecot-backup shell script which with adds some nice wrapping around dsync to loop over all the users mailboxes, tar up the files, prune older backups and send email alerts if errors occur.