EXT4 performance has become really bad on system with lots of small files

I have a small embedded device that has only 128MB of RAM

attached to this device is a 2TB USB2 hard disk

I've been very happy with the performance of the device up until recently when either the number of files has cross a threshold of the capacity of the disk has cross a threshold (I'm not sure which)

on the disk are many small files, due to the nature of the writing application files are organized in a very balanced way - no leaf node directory has more than 200 files and there are just over 800,000 files.

I'm hoping to get a lead on something to investigate. The disk performance has dropped significantly, the device was zipping along quite well and then all of a sudden performance dropped like a rock.

My assumption is that the organizational structure I've chosen on disk for my files has somehow hurt the inode caches ability to remain zippy.

as an experiment, I dismounted the disk (flushing caches, verified with free). Then from a command prompt I navigated deep into the directory structure. All told this directory (and its children) had only about 3200 files contained beneath it, and at this point 'free' showed >117MB of free memory

at this point, I typed the command 'find' followed by 'free'

'find' showed about 3000 files, but the memory usage went from ~117MB to ~2MB

I understand the balances of cache vs free memory, and how the kernel considers an empty page a bad page - however 115MB of cached content from a directory of 3000 files points to a serious gap in my understanding. I'm hoping someone will help me understand whats happening

can I assume a balanced tree is the way to go for having lots of files?

Solution 1:

Very good problem description.

Based on what you said, I think what you are seeing is slab usage going high. A good experiment would be to run a cat /proc/meminfo and cat /proc/slabinfo over a 3 second delay while you go deep into the fs hierarchy and discover the 3000 files. What essentially is happening is that the kernel will traverse the fs structure and scans the individual files and their inodes and all of them are stored in memory. If you check /proc/slabinfo you will see an object called ext4_inode_cache which tells you how much memory each inode will take. Multiply this with the no of objects (obj_size * no_obj) and you get the amount of memory used by the object. The deep you go into fs hierarchy, the more memory will be consumed until system hits the high watermark of the memory zone. At which point, kernel will start reclaiming.

If you poke into meminfo and slabinfo, you will get the details you are looking for. If you want me to look, pastebin it ;)