How to rescue a server with no free inodes (from DDOS)

One of my web server was DDOS attacked. All is well except there are millions of PHP session files used up 100% inodes of the partition. There is only one partition for the entire /

Tried several solutions, but only worked to some extend. https://unix.stackexchange.com/questions/37329/efficiently-delete-large-directory-containing-thousands-of-files?newreg=07f276292205457ab9975a0ea68e9273

http://www.slashroot.in/which-is-the-fastest-method-to-delete-files-in-linux

After freed up 8% of inodes, the disk become extremely slow to delete anything more.

rm -f filename* 

rsync -a --delete empty_dir/    yourdirectory/

perl -e 'for(<*>){((stat)[9]<(unlink))}'

And the disk look like this now

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/vda1      2621440 2385895 235545   92% /
tmpfs           128789       1 128788    1% /dev/shm

There are still 6million+ small files in a dir. The above methods delete at about 2 files/sec

I read about b-tree re-balancing. But how do I diagnose/solve the slow delete problem?

```

Solution 1:

A quick thing to do is to move/rename your current /tmp directory and create a new one so that normal usage of /tmp isn't impacted anymore.

mkdir /newtmp
chmod 1777 /newtmp
mv /tmp /tmp-old && mv /newtmp /tmp 

and maybe you need to do the same for /var/tmp as well. That allows your to peacefully clean up /tmp-old.

As a mitigating measure:

Consider using some of your memory to create a separate tempfs partition to use as storage for your PHP session files, which will somewhat limit the impact on the rest of your system.

i.e.

mkdir /var/cache/php
chmod 1777  /var/cache/php
mount -t tmpfs size=500M  /var/cache/php

and edit your php.ini and set

session.save_path = "/var/cache/php"