rdiff-backup and running out of disk space. Something weird's going on

I've got an rdiff-backup setup that backs up a largeish sql dump (1.6GB) over a network. The server it's backing up to (Ubuntu 10.10 server) has a small local disk (4GB, 1GB free) but has a cifs mount to a large nas (plenty of free space). I've a softlink set up from a user account's home dir to the NAS cifs mount, and the rdiff-backup job initiated from the client is writing to that directory.

Recently, the job has been failing. The logfile says:

Sun Oct 24 21:20:14 2010 Sending back exception [Errno 28] No space left on device of type type 'exceptions.IOError'

I've run the job manually with watch df running on the server, and I can see the 1GB of free space on the local drive diminishing until it hits zero. Then the job fails. To be clear, disk space is not an issue on the NAS.

So, for some reason, rdiff-backup is using somewhere on the local disk as a temp working directory rather than the actual directory where the backups are stored, on the NAS device, as I'd expect. However, I can't find which dir it's using so I can mount that separately with its own blob of space. Doing du -sx on / while the job is running reveals the total filesize of / remaining constant at 3.0GB, even though df says / is full.

If I list the file handles open by the rdiff process, I get a few sockets and some files on NAS mount. Nothing obvious on the local disk.

What's going on? Why does df say / is full, but du -sx say there's 1GB free? Why is rdiff-backup filling up the local disk when it should be using the cifs mount? What's it actually writing to?

Solution 1:

As maco already mentioned in comments, /tmp is the problem.

With rdiff-backup you can configure temp directory location with parameter

--tempdir path
          Sets  the  directory  that  rdiff-backup  uses for temporary files
          to the given path. The environment variables TMPDIR, TEMP, and TMP
          can also be used to set the temporary files directory. See the 
          documentation of the Python tempfile module for more information.

If you set tempdir to that CIFS mount, problem should go away. (We had same problem, and that fixed it, you may have something else too)