Strange processes on Server consume CPU

I have noticed 15% CPU load on the Server which is currently offline. It has mounted GlusterFS volume over TCP. Looking through the top, it shown me it's glusterfs. After that I tried to figure what exactly is using it and I got this:

# lsof /storage/
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF                NODE NAME
find    16433 nobody  cwd    DIR   0,19     8192 9259265867489333824 /storage/200000/200000/200700/200704/08

Then:

# ps uax | grep find
root     16415  0.0  0.0   4400   724 ?        SN   06:34   0:00 /bin/sh /usr/bin/updatedb.findutils
root     16423  0.0  0.0   4400   336 ?        SN   06:34   0:00 /bin/sh /usr/bin/updatedb.findutils
nobody   16431  0.0  0.0  39524  1376 ?        SN   06:34   0:00 su nobody -s /bin/sh -c /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)' \) -prune -o -print0
nobody   16432  0.0  0.0   4400   616 ?        SN   06:34   0:00 sh -c /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)' \) -prune -o -print0
nobody   16433  0.3  0.0  13612  1532 ?        SN   06:34   0:38 /usr/bin/find / -ignore_readdir_race ( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o -type d -regex \(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\) ) -prune -o -print0

I killed 16432 and 16433 and CPU now is %0 again.

Can someone tell me anything about these ugly find commands? Is it possible it's cause by the other Server which also have this /storage mounted?

According monitoring, it happens every day at the same time.

That looks like it's part of the daily updatedb job that runs to update the databases used by the locate command.

You'll probably find it in /etc/cron.daily as mlocate or similar.

If you use ps -ef you get the PID (process) and PPID (parent PID) which can be used to track back. You probably would have seen that the processes you killed had PPIDs 16415, 16423.

Tools like pstree are handy for this kind of thing too.

pstree -p -H5295

gives output like this

  |-sshd(5291)---sshd(5294)---bash(5295)-+-more(6098)
  |                                      `-pstree(6097)

As Iain noted, it's almost certainly updatedb(8). updatedb tries really hard to only index your local filesystems. On my machine, it's done by only including filesystems of type HFS and UFS. On your machine, it's done by specifically excluding various filesystem types like NFS, AFS, SMB and the like.

The problem with the exclusion approach, as you've noticed, is that when somebody creates a new network filesystem type - like GlusterFS for example - updatedb needs to be modified to exclude that type of filesystem as well.

For me, updatedb is a shell script, so I can trivially change the filesystem types. I would suspect that's true on your system as well. Assuming GlusterFS is of type glusterfs, you can probably just add another test:

-fstype glusterfs

in that line of tests for other filesystem types and be merrily on your way.

Or, of course, you could just turn off updatedb entirely if you never use the locate(1) command.