FreeNAS: "/var: write failed, filesystem is full"

Upon logging into our FreeNAS via shell, it says

/var: write failed, filesystem is full

So I did

> df -h
Filesystem             Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs2a    927M    358M    494M    42%    /
devfs                  1.0K    1.0K      0B   100%    /dev
/dev/md0               4.6M    1.8M    2.4M    43%    /etc
/dev/md1               824K    2.5K    756K     0%    /mnt
/dev/md2               149M    149M    -12M   109%    /var
/dev/ufs/FreeNASs4      20M    1.7M     17M     9%    /data
raid-5x3               100G     58M    100G     0%    /mnt/raid-5x3
raid-5x3/alpha          11T     11T    100G    99%    /mnt/raid-5x3/alpha
raid2                   11T    9.8T    903G    92%    /mnt/raid2

The biggest folder in /var was log ... and the biggest file in there was messages (146669553 bytes) so I rm messages.

But df still shows the same exact thing, and I'm still getting the error.

Solution 1:

This is most likely because the file that used to be known as /var/log/messages is still open despite having been unlinked.

Kill/restart the process that holds the file open (I would bet on syslogd) so that the space can be freed.

Alternatively there could be more links to the file, but that seems rather unlikely unless you have deliberately set that up.

Solution 2:

Whatever process writes to that file still has an open file handle for it. The space won't be reclaimed until that file handle is closed. Either restart the process (probably your syslog daemon) or restart the entire system if you don't care to track down the process.