Where are RAM snapshots stored by libvirt?

Solution 1:

The qcow2 format does provide permanent internal storage for the virtual machine memory. When snapshotting a running qcow2-backed virtual machines, its RAM state is dumped inside the very same qcow2 file used for storage.

From the official qcow2 page:

Each snapshot is described by a header:

  typedef struct QCowSnapshotHeader {
      /* header is 8 byte aligned */
      uint64_t l1_table_offset;

      uint32_t l1_size;
      uint16_t id_str_size;
      uint16_t name_size;

      uint32_t date_sec;
      uint32_t date_nsec;

      uint64_t vm_clock_nsec;

      uint32_t vm_state_size;
      uint32_t extra_data_size; /* for extension */
      /* extra data follows */
      /* id_str follows */
      /* name follows  */
  } QCowSnapshotHeader;

which further explains:

vm_state_size gives the size of the virtual machine state which was saved as part of this snapshot. The state is saved to the location of the original L1 table, directly after the image header.

On the other hand, what does happen when taking a snapshot of a raw image file (ie: not using qcow2)? Libvirt calls these kind of snapshot an external snapshot (compare it to the internal snapshot of a qcow2 file), because a new qcow2 file is generated and linked to the original (and now read-only) raw file. Libvirt then saves the virtual machine state (ie: its RAM content) in a specific file, generally inside a subdir of /var/lib/libvirt/. A metadata XML files is finally created to "glue" the dump file to the disk state saved in the overlay file.

Libvirt support for external snapshot is not very robust; for example (if things have not changed very recently) you can create an external snapshot with a single command but you need multiple complex command to delete/revert it. At the same time, for production workload RedHat officially recommend using external snapshots because they are somewhat more robust and faster than a long chain of internal snapshots (and they sell RHEV which hides the complexity of external snapshots away from the user).