Backing up volumes from host vs from the inside of the container

The official docker documentation suggests as a good practice to backup the entire docker volume as it is.

When you need to back up, restore, or migrate data from one Docker host to another, volumes are a better choice. You can stop containers using the volume, then back up the volume’s directory (such as /var/lib/docker/volumes/).

At the same time the docs state that one should must not modify volumes under no circumstance. And restoring a volume manually into the /var/lib/docker/volumes/ violates that warning. So these two statements seem to be contradictory to me.

Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem.

Questions:

• Is it really safe to backup and restore volumes in that fashion between different versions of docker runtime?
• Do volumes have some inner structure that is dependent on the runtime version? Or are they just regular files?
• Is it safer to mount the volume to some container and backup it's data from the inside of the container?

This is a complex question and as these things go, the answer is "it depends". The primary factors that relate to the question are the filesystem driver and the volume of activity within the docker daemon's scope. Assuming a "vanilla" situation with low volume writes, you should be relatively safe backing up /var/lib/docker directly; however, this is equivalent to backing up any running system that manages transient state (risks could be unfinished writes, for example).

Ultimately, processes running in a container context are still regular processes... so, it depends on the contained app, the frequency of disk writes, and the filesystem driver itself.

From an operational perspective, stopping the container or the main writer process within it would be prudent before backing it up.