rsync : write failed - No space left on device (28) despite using --inplace

I am using a script to make a backup of a running Debian Stretch based Linux Distro (NextCloudPi)

The method in the script that does the backup uses rsync.

backup()
{
  mntimg
  sync
  rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/lost+found' --exclude '/media' --exclude '/mnt' \
--exclude '/proc' --exclude '/run' --exclude '/sys' --exclude '/tmp' --exclude '/var/swap' --exclude '/etc/udev/rules.d/70-persistent-net.rules' \
--exclude '/var/lib/asterisk/astdb.sqlite3-journal' "${OPTIONS[@]}" / "${MNTPATH}/"
..
..
}

When I run the script I point it to save the backup .img file to an externally attached USB-HDD drive.

This drive is EXT4 formatted and is mounted.. I can browse it from Manjaro file explorer. This is writable and has 2.3TB free space.

The backup file is going to be about 7.8GB and I have got 22GB free space on the rootfs (/) on the SD Card that I am backing up.

Each time I run the script I get an error rsync: write failed on.. no space left on device:

root@NEXTCLOUDPI:~# image-backup

Image file to create? /media/4TB2/nextcloudpi18Nov2021v3.img

Initial image file ROOT filesystem size (MB) [7526]? 7800

Added space for incremental updates after shrinking (MB) [0]? 

Create /media/4TB2/nextcloudpi18Nov2021v3.img (y/n)? y

Starting full backup (for incremental backups, run: /usr/local/bin/image-backup /media/4TB2/nextcloudpi18Nov2021v3.img)
rsync: write failed on "/tmp/img-backup-mnt/usr/src/linux-headers-4.14.93-Re4son-v7+/include/linux/suspend.h": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]

Unable to create backup

root@NEXTCLOUDPI:~#

I still get the problem even if I add in the rsync option --inplace so that did not solve my problem.

I did a sudo du -sh /usr/src and the size is 150MB.
I've got 37,000 files and 12,000 sub-folders in /usr/src so I was thinking maybe I am running out of inodes but... I did a df -i and my inode usage is 14% in the root dir (/).

The problem seems to happen close to the end.. in this case a 7.9GiB file is created. I tried flashing that to an sd card with etcher but it didn't boot.

Any ideas what is going wrong here? I have enough space on the rootfs for rsync to save things to /tmp if it needs to. But even when I use the --inplace option it still says: rsync: write failed on "/tmp/... blah blah... No space left on device (28)

Solution 1:

Now based on your recent comment I can say that most likely it just run our of inodes. When you create ext4 fs by default mkfs allocates inodes based on partition/image size. So the less is the image size, the less is the inode count. You edit this script to allocate more inodes, just find a line where it does mkfs.ext4 and change the number of inode it allocates with -i or -N option.

Find this line in image-backup:

    mkfs.ext4 -q -b 4096 "${LOOP}p2" > /dev/null

and change it to:

    mkfs.ext4 -q -b 4096 -i 4096 "${LOOP}p2" > /dev/null

This will make 4x more inodes than the default settings.

I don't think something in your server can produce such effect. While sparse files might be the reason too, you can mitigate it with -S rsync option which will handle sparse files correctly. But it conflicts with --inplace option.