How do I use OverlayFS?
This answer and email message indicate that something called "OverlayFS" is available in Ubuntu 11.10 and will forcefully replace aufs in Ubuntu 12.04.
How do I use it? Where is its documentation?
Edit: Since writing this answer, some things have changed in overlayfs, namely the addition of a required parameter workdir, see totti's answer below for a detailed description of this new parameter.
I finally managed to find it. I found references to it in the kernel source, but for some reason it doesn't appear in the git tree on kernel.org. But! If you pull the Ubuntu kernel source like this: apt-get source linux-image-3.0.0-16-generic you can find it in linux-3.0.0/Documentation/overlayfs.txt. It is also available in the linux-doc package in /usr/share/doc/linux-doc/filesystems/overlayfs.txt.gz.
As the actual help documentation is more of a "how it works" instead of a "how to mount with it," here's a brief rundown (there is one example in the kernel documentation):
mount -t overlayfs -o [mount options] overlayfs [mountpoint for merged system]
Where [mount options] can be:
- lowerdir=somedir: lowerdir is the directory you're going to lay your new filesystem over, if there are duplicates these get overwritten by (actually, hidden in favor of) upperdir's version
- upperdir=somedir: upperdir is the directory you want to overlay lowerdir with. If duplicate filenames exist in lowerdir and upperdir, upperdir's version takes precedence.
- standard mount options. The only one I've seen from code is ro/rw, but you can experiment.
One thing that confused me at first, so I should probably clarify, is that mounting an overlayfs does not actually mount a filesystem. I was trying to mount a squashfs filesystem using an overlayfs mount, but that's not how it works. You must first mount the (in my case squashfs) filesystem to an arbitrary directory, then use overlayfs to merge the mount point (a directory) and another directory onto a tertiary directory (the overlayfs mount point)(edit: this "tertiary" directory can actually be the upperdir= directory). The tertiary directory is where you will see the merged filesystems (or directory trees - it's flexible).
Example 1, overlaying the root filesystem
I've been working on an Ubuntu hybrid boot disk where the base Ubuntu system exists as filesystem.squashfs and I have files called ubuntu.overlay kubuntu.overlay xubuntu.overlay and lubuntu.overlay. The .overlay files are base installs of said systems with the contents of filesystem.squashfs pruned (to save space). Then I modified the init scripts to overlay the correct distro's .overlay file (from a boot parameter) using overlayfs and the above options and it works like a charm!
These are the lines that I used in my init scripts (once all variables are translated):
mkdir -p /overlay
mount -t squashfs /cdrom/casper/ubuntu.overlay /overlay
mount -t overlayfs -o lowerdir=/filesystem.squashfs,upperdir=/overlay overlayfs /
Note that filesystem.squashfs above is a directory created by casper, not a file.
These three statements create an /overlay directory, mount a squashfs filesystem on the /overlay directory and then use OverlayFS to essentially merge the contents of /overlay over /.
Example 2, transparent merging of two directories
In the process of re-building my live USB for each release, I use OverlayFS to save a bunch of time. I start off with a directory called ubuntu-base which contains the contents of the ubuntu-core image which is the most basic install. I will then create directories called ubuntu, kubuntu, lubuntu, and xubuntu.
Then, I use OverlayFS to make the files from the ubuntu-base show up in the individual directories. I would use something like this:
mount -t overlayfs -o lowerdir=ubuntu-base,upperdir=kubuntu overlayfs kubuntu
This makes the files from ubuntu-base show up in the kubuntu folder. Then, I can chroot to the kubuntu folder and do something like apt-get install kubuntu-desktop. Any changes that are made while in this OverlayFS mount will remain in the upper directory, in this case the kubuntu folder. Then, once I unmount the OverlayFS mount the files that really exist in ubuntu-base but are "mirrored" into the kubuntu folder vanish unless they have been changed. This keeps me from having to have multiple copies of the files in ubuntu-base while still being able to use them as if they physically exist in each location.
From https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt:
Upper and Lower
An overlay filesystem combines two filesystems - an 'upper' filesystem and a 'lower' filesystem. When a name exists in both filesystems, the object in the 'upper' filesystem is visible while the object in the 'lower' filesystem is either hidden or, in the case of directories, merged with the 'upper' object.
It would be more correct to refer to an upper and lower 'directory tree' rather than 'filesystem' as it is quite possible for both directory trees to be in the same filesystem and there is no requirement that the root of a filesystem be given for either upper or lower.
The lower filesystem can be any filesystem supported by Linux and does not need to be writable. The lower filesystem can even be another overlayfs. The upper filesystem will normally be writable and if it is it must support the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses, so NFS is not suitable.
A read-only overlay of two read-only filesystems may use any filesystem type.
Directories
Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.
Where both upper and lower objects are directories, a merged directory is formed.
At mount time, the two directories given as mount options "lowerdir" and "upperdir" are combined into a merged directory:
mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /mergedThe "workdir" needs to be an empty directory on the same filesystem as upperdir.
Then whenever a lookup is requested in such a merged directory, the lookup is performed in each actual directory and the combined result is cached in the dentry belonging to the overlay filesystem. If both actual lookups find directories, both are stored and a merged directory is created, otherwise only one is stored: the upper if it exists, else the lower.
Only the lists of names from directories are merged. Other content such as metadata and extended attributes are reported for the upper directory only. These attributes of the lower directory are hidden.
I have extended these artikels to include a Script for overlayfs that sets up a read-only root fs.
- English: https://help.ubuntu.com/community/aufsRootFileSystemOnUsbFlash
- German: http://wiki.ubuntuusers.de/Nur-Lesen_Root-Dateisystem
Hope it helps.
Minimal runnable example
# Create the filesystems.
dd if=/dev/zero of=lower.ext4 bs=1024 count=102400
mkfs -t ext4 lower.ext4
cp lower.ext4 upper.ext4
mkdir lower upper overlay
sudo mount lower.ext4 lower
sudo mount upper.ext4 upper
sudo chown "$USER:$USER" lower upper
printf lower-content > lower/lower-file
# Upper and work must be on the same filesystem.
mkdir upper/upper upper/work
printf upper-content > upper/upper/upper-file
# Work must be empty. E.g. this would be bad:
#printf work-content > upper/work/work-file
# Make the lower readonly to show that that is possible:
# writes actually end up on the upper filesystem.
sudo mount -o remount,ro lower.ext4 lower
# Create the overlay mount.
sudo mount \
-t overlay \
-o lowerdir=lower,upperdir=upper/upper,workdir=upper/work \
none \
overlay \
;
# Interact with the mount.
printf 'overlay-content' > overlay/overlay-file
ls lower upper/upper upper/work overlay
# Write to underlying directories while mounted
# gives undefined behaviour.
#printf lower-content-2 > lower/lower-file-2
#printf upper-content-2 > upper/upper-file-2
# Unmount the overlay and observe state.
sudo umount overlay
ls lower upper/upper upper/work
# Cleanup.
sudo umount upper lower
GitHub upstream.
Output of the first ls with the mount:
lower:
lost+found lower-file
overlay:
lost+found lower-file overlay-file upper-file
upper/upper:
overlay-file upper-file
upper/work:
work
Output of the second ls without the mount:
lower:
lost+found lower-file
upper/upper:
overlay-file upper-file
upper/work:
work
Interpretation:
- lower: was unchanged after writing to overlay
- upper: received the modification to overlay
- overlay: shows files from both upper and lower
- work: contains some random content (a
work/directory) we should not care about
Example adapted from: Example OverlayFS Usage
Here is a more complex example with multiple lower layers: Overlayfs reload with multiple layers (migration away from aufs)
Tested on Ubuntu 18.04, Linux kernel 4.15.0.