Where is my space on /var being used?
Solution 1:
You can run lsof | grep deleted
and check witch programs allocated this space (and the deleted file).
example:
[root@mab-01 ~]# lsof | grep deleted
hald-addo 2651 haldaemon txt REG 253,0 15720 3769183 /usr/libexec/hald-addon-keyboard.#prelink#.IhBW5L (deleted)
yum-updat 2899 root txt REG 253,0 4736 3276902 /usr/bin/python.#prelink# (deleted)
mongod 5535 mongod txt REG 253,0 8640360 3484794 /usr/bin/mongod (deleted)
mongod 5535 mongod 1w REG 253,0 278032 262244 /var/log/mongo/mongod.log.rpmsave (deleted)
mongod 5535 mongod 2w REG 253,0 278032 262244 /var/log/mongo/mongod.log.rpmsave (deleted)
Solution 2:
/var/log did fill up
If you delete log files that are open for writing by a process, the filenames dissapear (so are not seen by du?) but the space allocated is still allocated and as the process continues to write, the allocated space can increase.
If the logs were TomCat logs you need to tell Tomcat to reopen it's log files.
Note "copytruncate" in this example. I don't know if this applies to your situation though.