Are ZFS snapshots really "free"?

Solution 1:

I did a test, actually for an un-related problem (zfs send being unable to send a filesystem with over 42,000 snapshots) and to my surprise, I discovered that zfs snapshots actually consume a handful of megabytes. In my tests, this was about 4 MiB/snapshot.

To test this, I created an empty zpool with a single filesystem with no files, directories, or other data at all. I then attempted to create 100,000 snapshots and to my surprise, this faied after creating only 50,698 snapshots:

root@test:~# zpool create tank nvme-nvme.15ad-564d57617265204e564d455f30303030-564d77617265205669727475616c204e564d65204469736b-00000002
root@test:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   199G   116K   199G        -         -     0%     0%  1.00x    ONLINE  -
root@test:~# zfs create tank/test
root@test:~# zfs list
NAME        USED  AVAIL     REFER  MOUNTPOINT
tank        118K   193G       24K  /tank
tank/test    24K   193G       24K  /tank/test
root@test:~# ls -lR /tank/
/tank/:
total 1
drwxr-xr-x 2 root root 2 Mar 14 15:16 test

/tank/test:
total 0
root@test:~# find /tank
/tank
/tank/test
root@test:~# for i in {0..100000}; do zfs snapshot tank/test@snap$i; done
cannot create snapshots : out of space
cannot create snapshots : out of space
...
cannot create snapshots : out of space
cannot create snapshots : out of space
root@proxmoxtest:~# zfs list -t snapshot | wc -l
50698
root@proxmoxtest:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   200G   193G  6.23G        -         -    84%    96%  1.00x    ONLINE  -

Based on this test, it seems that ZFS needs about 3.9 MiB of space for storing metadata for each snapshot on a 200GiB zpool. This appears to vary by pool size; when I tested with a 20GiB zpool, it came out to ~1.8 MiB/snapshot.

So while zfs snapshots may use a "negligible' amount of space (definition of "negligible" may be opinion-based) that cost is not zero; there is a small space overhead (measurable in megabytes) for each snapshot, even when no blocks change.