where are kvm-snapshots stored?

I run several VM's on KVM/ubuntu, which all get started with the -snapshot parameter (the VM's only compute several things which can be destroyed after reboot).

Out of the docs I've read, the changes are not written back to the image, but stored in temp-files and these are deleted after shutdown.

Now, I wonder where on the filesystem these "temp-files" are stored?

Try use lsof to find snapshot storage:

# ps aux | grep kvm
$PID ?        Sl   5289:40 /usr/bin/kvm ...
# lsof -n -p $PID
COMMAND   PID USER   FD   TYPE   DEVICE        SIZE     NODE NAME

or one command:

# lsof -n -p $(ps ax | grep [k]vm | awk '{print $1}' | head -1)

I just wondered myself where "-snapshot" stores its changes, and eventually found out where: It seems to use the same algorithm as "mktemp".

That is, if you set the environment variable TMPDIR to the path of some directory, then it stores its temporary data there.

Otherwise, it stores them in /tmp.

Therefore, if you want to save temporary snapshots in /var/tmp/kvm rather than in /tmp, run qemu as follows:

$ TMPDIR=/var/tmp/kvm qemu-system-x86_64 -enable_kvm ...

Alternatively, you can "export" the variable first:

$ export TMPDIR=/var/tmp/kvm
$ qemu-system-x86_64 -enable_kvm ...

This might be better if you start multiple VMs, because you don't need to specify the variable for every invocation.

Also, remember to also add "-no-shutdown" to the qemu options if you want to be able to keep the changes rather than losing them when the VM shuts down.

With -no-shutdown, the vm enters a stopped state after shutdown, but the vm process stays alive.

This allows you to connect to the KVM monitor console and issue the "commit" command, which writes all changes from the temporary file back to the VM image file. After that, you issue the "quit" command which actually terminates the VM process.

Without "-no-shutdown", you do not have the chance to issue the "commit" command, and without it all changes in the temporary file are lost once the VM process shuts down.

And another note: Do not wonder that no file shows up in the directory specified by $TMPDIR once the VM has started with "-snapshot".

The file is there, but qemu deletes the file immediately after it has created it, and therefore it does not show up in a directory listing.

But the qemu process still holds a reference to the file, and so the file is kept alive until the qemu process terminates. Only then the "deleted" file will be truly deleted, and the space occupied by it will be returned as free space to the filesystem.