Docker volume error read-only file system\\\"\"": unknown

Context

I am experiencing something strange while integrating promtail to my stack

ERROR: for asmodius_promtail  Cannot start service promtail: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/asmodius_nginx-logs/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged\\\" at \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx\\\" caused \\\"mkdir /var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx: read-only file system\\\"\"": unknown

The volume mentioned in this error is the one I use so promtail could read Nginx logs. (It only reads the files and does not need to write anything in the dir) Here is a portion of the docker-compose.yml

version: '3.7'
services:
  nginx: 
    container_name: ${SERVER_NAME}_nginx
    restart: always
    networks: 
      - monitoring
      - web
    image: nginx:1.17.5-alpine
    entrypoint: /entrypoint.sh
    ports: 
      - 80:80
      - 443:443
    volumes: 
      - ./nginx/site.conf:/etc/nginx/conf.d/site.conf
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/entrypoint.sh:/entrypoint.sh
      - nginx-logs:/var/log/nginx
      - certificates:/etc/letsencrypt/

  promtail: 
    restart: always
    container_name: ${SERVER_NAME}_promtail
    image: grafana/promtail:master-b652f0a
    command: -config.file=/etc/promtail/docker-config.yaml
    volumes: 
      - nginx-logs:/var/log/nginx
      - /var/log:/var/log:ro
      - ./promtail/docker-config.yaml:/etc/promtail/docker-config.yaml:ro
    networks: 
      - monitoring

and the volume "inspection"

    {
        "CreatedAt": "2020-04-28T20:08:41Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "asmodius",
            "com.docker.compose.version": "1.24.1",
            "com.docker.compose.volume": "nginx-logs"
        },
        "Mountpoint": "/var/lib/docker/volumes/asmodius_nginx-logs/_data",
        "Name": "asmodius_nginx-logs",
        "Options": null,
        "Scope": "local"
    }

Strangely enough, the very same docker-compose.yml works like a charm on my Debian 9

• Docker version 19.03.8, build afacb8b7f0
• docker-compose version 1.22.0, build f46880fe

but fails on CentOS

• Docker version 19.03.8, build afacb8b
• docker-compose version 1.24.1, build 4667896b

Leads

I don't quite believe that docker-compose version could be the problem. I tried the following

• stop the entire stack, rm nginx-logs volume and start again
• use :ro not use :ro

Questions

Since I do not entirely understand the ERROR message, an explanation of the following elements might help me figure what I am doing wrong.

• read-only file system\\\"\"": unknown: I just don't get what "\" is
• why does it mention a merge ? A merge with what?
• mkdir.... so it tries to create a folder somewhere in /var/lib/docker. But why does it have to do so for this specific volume while all the others (and there is quite a lot of them) succeed every time I docker-compose up -d --force-recreate (for it is how I add/update services on this machine)

All insights are welcome

mkdir /var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx: read-only file system\\\"\"

This is complaining that it cannot create a directory under /var/lib/docker. None of that filesystem should be read-only. Perhaps you didn't start the docker engine as root, or perhaps there's something wrong with the filesystem itself. I'd double check that you haven't run out of capacity, either in free bytes or free inodes, with:

df /var/lib/docker/.
df -i /var/lib/docker/.

If you are running Docker Desktop, then as a last resort, destroy and recreate the docker VM. This will result in losing all the volume data, built images, running cntainers, etc, so backup anything you want to save.