Utility says there's no more free disk space, when there's plenty
Somehow my system is in a pickle.
$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
linux-headers-3.2.0-29 linux-headers-3.2.0-29-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
linux-headers-3.2.0-80
The following NEW packages will be installed
linux-headers-3.2.0-80
0 to upgrade, 1 to newly install, 0 to remove and 251 not to upgrade.
3 not fully installed or removed.
Need to get 0 B/11.7 MB of archives.
After this operation, 56.4 MB of additional disk space will be used.
Do you want to continue [Y/n]?
(Reading database ... 1255202 files and directories currently installed.)
Unpacking linux-headers-3.2.0-80 (from .../linux-headers-3.2.0-80_3.2.0-80.116_all.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-80_3.2.0-80.116_all.deb (--unpack):
unable to create `/usr/src/linux-headers-3.2.0-80/include/linux/sunrpc/gss_err.h.dpkg-new' (while processing `./usr/src/linux-headers-3.2.0-80/include/linux/sunrpc/gss_err.h'): No space left on device
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-3.2.0-80_3.2.0-80.116_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
But:
us@desktop:/var/log$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 20158332 16166516 2967816 85% /
udev 4026512 4 4026508 1% /dev
tmpfs 807064 996 806068 1% /run
none 5120 0 5120 0% /run/lock
none 4035308 128 4035180 1% /run/shm
/dev/sda4 904990760 51533960 807485908 6% /home
Stumped!
us@desktop:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1281120 1278499 2621 100% /
udev 1006628 507 1006121 1% /dev
tmpfs 1008827 445 1008382 1% /run
none 1008827 3 1008824 1% /run/lock
none 1008827 6 1008821 1% /run/shm
/dev/sda4 57466880 110192 57356688 1% /home
Oh so the inode table is full. Never had that before, what causes that then?
As shown in the output of df -i, you are running very low on number of free inodes on / and hence the dpkg fails to complete the operation.
Most of the time, this is caused by numerous small files created somewhere (unintentionally) in the filesystem.
You can check the files of size less than 1 KB and start removing them if not needed:
sudo find / -type f -size -1k -ls
Similarly you can check with increasing file sizes until you get to the root of the problem.
All credit to @heemayl for the pointers that resulted in this answer. I've accepted that answer, but here's the details of how I fixed it, in case it's useful to others.
The issue was caused by a full inode table. As I understand, there are two parts to a filesystem, inodes and space. inodes are related to the number of files, where as space is just how big they are. My system was chock-full of millions of tiny files caused by Ubuntu not removing old kernels. As the system has run for a long time I had over 40 kernels, incl. header files, installed which is too much for a 20Gb system partition's file system to take.
This meant dpkg could not complete an installation - it needed more space to complete its work - but unfortunately that meant neither could I create space by using apt to uninstall things!
The solution I found was to move all the /usr/src/ files to my other partition, leaving just a symlink to the files on the full fs:
sudo mv /usr/src /home/usr-src
sudo ln -s /home/usr-src /usr/src
After doing this I was able to let apt complete:
sudo apt install -f
And after that I was able to uninstall all the older kernels that I did not need, freeing up space, and then I could move the src folder back.