'inode table usage' spiking every morning at 8am

Solution 1:

On a default ubuntu/debian set-up, some things will run via 'anacron'. The timing of this can depend on a number of factors, particularly when the machine started up, but on a server machine it will trigger at 7:30 a.m. every morning by default.

This is configured in the file /etc/cron.d/anacron , so a simple solution is to edit that file and and set the configured time to something more convenient (e.g. middle of the night if you are concerned about the tasks impacting day time performance) There is no need to restart anything.

The cron log which I enabled, was in fact showing the initiation of anacron at 7:30. Anacron delays for a little while before doing anything and also presumably queued up several jobs, the result being that one job was starting up at about 8a.m. each day and causing a lot of inode table usage (people have suggested the mlocate db re-index) I don't know that it was causing any performance problems, but it seems healthier now to see it doing this in the middle of the night.

There is some brief documentation specific to ubuntu here: https://help.ubuntu.com/community/CronHowto#How_Anacron_is_Set_Up

...and much more detailed explanation of the interplay between cron and anacron here: https://linuxformat.com//tuxradarchive/content/automate-linux-cron-and-anacron

Solution 2:

Look for cronjobs running at about that time. In Ubuntu check the files in /etc/cron.daily first, as well as root's crontab ("sudo crontab -l"). Most likely it is something like log rotation, or log analysis that is creating many temporary file.