Where is my space on /var being used?

Solution 1:

You can run lsof | grep deleted and check witch programs allocated this space (and the deleted file).

example:

[root@mab-01 ~]# lsof | grep deleted
hald-addo  2651 haldaemon  txt       REG              253,0      15720    3769183 /usr/libexec/hald-addon-keyboard.#prelink#.IhBW5L (deleted)
yum-updat  2899      root  txt       REG              253,0       4736    3276902 /usr/bin/python.#prelink# (deleted)
mongod     5535    mongod  txt       REG              253,0    8640360    3484794 /usr/bin/mongod (deleted)
mongod     5535    mongod    1w      REG              253,0     278032     262244 /var/log/mongo/mongod.log.rpmsave (deleted)
mongod     5535    mongod    2w      REG              253,0     278032     262244 /var/log/mongo/mongod.log.rpmsave (deleted)

Solution 2:

/var/log did fill up

If you delete log files that are open for writing by a process, the filenames dissapear (so are not seen by du?) but the space allocated is still allocated and as the process continues to write, the allocated space can increase.

If the logs were TomCat logs you need to tell Tomcat to reopen it's log files.

Note "copytruncate" in this example. I don't know if this applies to your situation though.