Creating a grow-on-demand encrypted volume with LUKS
I am trying to create an encrypted, growing-as-needed file system in with Linux. I am familiar with LUKS and cryptsetup.
I can create an empty file:
fallocate -l 512M /root/image
I can create a LUKS container on it:
cryptsetup -y luksFormat /root/image
And then "open" it:
cryptsetup luksOpen /root/image luksvolume
At this point, I can just make a file system on it:
mkfs.ext4 -j /dev/mapper/luksvolume
This is all fine and dandy. However, it doesn't address the "grow-on-demand" part of the question.
The idea is that copying a 2Gb file on the encrypted file system will "expand" the image so that it's big enough to contain the file.
Is it even possible to do?
Yes! It looks like it's possible. Let's check how it can be achieved. Note that this doesn't create a true grow-on-demand filesystem, as when the filesystem reaches the max size of the sparse file, it will report 'out of space' errors if more data still needs to be written.
Initially, I was investigating Thin Provisioning, a well-known technology to save storage space in virtualization scenarios. Unfortunately, in common Linux use-cases, it seems to be available only with LVM. As this seems a bit outside the scope of your question, I searched for something else.
The second concept I investigated is Sparse File. This is exactly suited to your question and... my initial doubt was: "OK. I can create a Sparse File. But what happens when I initialize it as a LUKS container? Will such initialization allocate all the available space? If not, what will happen when I will initialize the filesystem in such a container? Will an mkfs.ext4 allocate all the available space?". As I had no answer, I decided to try. So, let's see what happened.
Let's start from my current system, where I have only 3.3G of free space within the /repository filesystem:
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 258G 3,3G 99% /repository
Let's create a 10G sparse-file within such a filesystem, with:
root@iMac-Chiara:~# dd of=/repository/file_container.img bs=1G count=0 seek=10
0+0 record dentro
0+0 record fuori
0 byte (0 B) copiati, 0,000119606 s, 0,0 kB/s
and let's verify that... it's really a sparse file:
root@iMac-Chiara:~# ls -lh /repository/file_container.img
-rw-r--r-- 1 root root 10G dic 12 19:48 /repository/file_container.img
OK. So we have a 10G file, in a filesystem that previously had 3.3G of free space. How much free space still I have?
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 258G 3,3G 99% /repository
Still 3.3G. Nice. Sparse-file are really... sparse-file ;-) Let's step ahead, by creating a LUKS container within such a 10G file and... let's see if we run out of space:
root@iMac-Chiara:~# losetup /dev/loop0 /repository/file_container.img
root@iMac-Chiara:~# cryptsetup -y luksFormat /dev/loop0
WARNING!
========
Ciò sovrascriverà i dati in /dev/loop0 in modo irreversibile.
Are you sure? (Type uppercase yes): YES
Inserire la passphrase LUKS:
Verify passphrase:
root@iMac-Chiara:~# cryptsetup luksOpen /dev/loop0 secretfs
Inserire la passphrase per /dev/loop0:
root@iMac-Chiara:~#
So now I have an opened secrets container defined on top of my 10G sparse-file stored in a filesystem having only 3.3G of free space.
How much free space still I have?
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 258G 3,3G 99% /repository
Wonderful! Still 3.3GB. Our encrypted container required mostly no space!
Let's check if everything is OK or if there are something strange with our setup:
root@iMac-Chiara:~# cryptsetup status secretfs
/dev/mapper/secretfs is active.
type: LUKS1
cipher: aes-cbc-essiv:sha256
keysize: 256 bits
device: /dev/loop0
loop: /repository/file_container.img
offset: 4096 sectors
size: 20967424 sectors
mode: read/write
Everything seems OK so let's start using such a container to store something. Let's begin by creating an EXT4 filesystem inside it:
root@iMac-Chiara:~# mkfs.ext4 /dev/mapper/secretfs
mke2fs 1.42.5 (29-Jul-2012)
Etichetta del filesystem=
OS type: Linux
Dimensione blocco=4096 (log=2)
Dimensione frammento=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2620928 blocks
131046 blocks (5.00%) reserved for the super user
Primo blocco dati=0
Maximum filesystem blocks=2684354560
80 gruppi di blocchi
32768 blocchi per gruppo, 32768 frammenti per gruppo
8192 inode per gruppo
Backup del superblocco salvati nei blocchi:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: fatto
Scrittura delle tavole degli inode: fatto
Creating journal (32768 blocks): fatto
Scrittura delle informazioni dei superblocchi e dell'accounting del filesystem: fatto
root@iMac-Chiara:~#
It looks like it worked, as there was no track of "out of space". Let's check:
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 258G 3,2G 99% /repository
Uhm.... so something happened. We lost something like 100M of space but.... it's an expected behaviour: the creation of the EXT4 filesystem DO require the writing of lots of metadata. So it's normal that some space have been used by the creation process.
Is it a "working" EXT4 filesystem?
root@iMac-Chiara:~# tune2fs -l /dev/mapper/secretfs
tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: e63321c3-cee7-478d-a6af-cbdcaf1be1f7
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 655360
Block count: 2620928
Reserved block count: 131046
Free blocks: 2541265
Free inodes: 655349
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 639
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Sat Dec 12 19:58:05 2015
Last mount time: n/a
Last write time: Sat Dec 12 19:58:05 2015
Mount count: 0
Maximum mount count: -1
Last checked: Sat Dec 12 19:58:05 2015
Check interval: 0 (<none>)
Lifetime writes: 131 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: c8b3bf1b-9f05-4267-85d3-2ecfdbaa6dc3
Journal backup: inode blocks
Yes! It looks ok.
So now we have an EXT4 filesystem written inside an opened LUKS container defined on top of a 10G sparse-file stored within a 3.3G filesystem.
Let's see if everything works correctly, by allocating space "on-demand".
Let's start by writing 500M of dummy data to the encrypted FS
root@iMac-Chiara:~# mkdir /mnt/temp
root@iMac-Chiara:~# mount /dev/mapper/secretfs /mnt/temp
root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/random_data.bin bs=1M count=512
512+0 record dentro
512+0 record fuori
536870912 byte (537 MB) copiati, 2,35214 s, 228 MB/s
root@iMac-Chiara:~#
Have we been succesfull in creating the file?
root@iMac-Chiara:~# ls -lh /mnt/temp/random_data.bin
-rw-r--r-- 1 root root 512M dic 12 20:09 /mnt/temp/random_data.bin
It looks so.
What happened to our real filesystem?
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 259G 2,5G 100% /repository
Uau! We "lost" sligthly more than 500M. That's good, BTW, as the physical space is really allocated on demand!
Let's store another 2GB file:
root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/another_random_data.bin bs=1G count=2
2+0 record dentro
2+0 record fuori
2147483648 byte (2,1 GB) copiati, 25,6539 s, 83,7 MB/s
root@iMac-Chiara:~#
What happened?
root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 2,6G
-rw-r--r-- 1 root root 512M dic 12 20:09 random_data.bin
drwx------ 2 root root 16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29 2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:12 .
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 261G 484M 100% /repository
root@iMac-Chiara:~#
Really nice. What happens if we delete a file?
root@iMac-Chiara:~# rm /mnt/temp/random_data.bin
root@iMac-Chiara:~# sync
root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 2,1G
drwx------ 2 root root 16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29 2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:14 .
root@iMac-Chiara:~# df -h /repository
File system Dim. Usati Dispon. Uso% Montato su
/dev/sda3 275G 261G 484M 100% /repository
root@iMac-Chiara:~#
As expected, with sparse-file the behaviour is exactly like thin-provisioning: once allocated, storage-space cannot be claimed back when file are deleted. But this, in general, is OK. Don't you?
So at this point, the answer to your question should be complete. Right?
Addition:
Let's see what happens when the underlining storage gets full:
root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/a_third_random_data.bin bs=1G count=2
2+0 record dentro
2+0 record fuori
2147483648 byte (2,1 GB) copiati, 26,7142 s, 80,4 MB/s
root@iMac-Chiara:~#
What? it looks like it succeded! How this have been possible? Let's check!
root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 4,1G
drwx------ 2 root root 16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:17 a_third_random_data.bin
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29 2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:17 .
root@iMac-Chiara:~#
Uhm... It looks ok. Are we sure?
root@iMac-Chiara:~# df /repository
File system 1K-blocchi Usati Disponib. Uso% Montato su
/dev/sda3 288110208 275070448 0 100% /repository
we have run out of space! Without any error!
Even if it would be nice to investigate what really happened... I'm going to leave this to your curiosity and/or troubleshooting skills of other ServerFault members ;-)
Have fun!
BTW: I've tested all of the above, here:
root@iMac-Chiara:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.04
DISTRIB_CODENAME=raring
DISTRIB_DESCRIPTION="Ubuntu 13.04"
root@iMac-Chiara:~# uname -r
3.8.0-31-generic
root@iMac-Chiara:~# dpkg -l cryptsetup-bin
[...]
ii cryptsetup-bin 2:1.4.3-4ubuntu2 amd64 disk encryption support - command line tools
root@iMac-Chiara:~#