Caching/preloading files on Linux into RAM
I have a rather old server that has 4GB of RAM and it is pretty much serving the same files all day, but it is doing so from the hard drive while 3GBs of RAM are "free".
Anyone who has ever tried running a ram-drive can witness that It's awesome in terms of speed. The memory usage of this system is usually never higher than 1GB/4GB so I want to know if there is a way to use that extra memory for something good.
- Is it possible to tell the filesystem to always serve certain files out of RAM?
- Are there any other methods I can use to improve file reading capabilities by use of RAM?
More specifically, I am not looking for a 'hack' here. I want file system calls to serve the files from RAM without needing to create a ram-drive and copy the files there manually. Or at least a script that does this for me.
Possible applications here are:
- Web servers with static files that get read alot
- Application servers with large libraries
- Desktop computers with too much RAM
Any ideas?
Edit:
- Found this very informative: The Linux Page Cache and pdflush
- As Zan pointed out, the memory isn't actually free. What I mean is that it's not being used by applications and I want to control what should be cached in memory.
vmtouch seems like a good tool for the job.
Highlights:
- query how much of a directory is cached
- query how much of a file is cached (also which pages, graphical representation)
- load file into cache
- remove file from cache
- lock files in cache
- run as daemon
vmtouch manual
EDIT: Usage as asked in the question is listed in example 5 on vmtouch Hompage
Example 5
Daemonise and lock all files in a directory into physical memory:
vmtouch -dl /var/www/htdocs/critical/
EDIT2: As noted in the comments, there is now a git repository available.
This is also possible using the vmtouch Virtual Memory Toucher utility.
The tool allows you to control the filesystem cache on a Linux system. You can force or lock a specific file or directory in the VM cache subsystem, or use it to check to see what portions of a file/directory are contained within VM.
How much of the /bin/ directory is currently in cache?
$ vmtouch /bin/
Files: 92
Directories: 1
Resident Pages: 348/1307 1M/5M 26.6%
Elapsed: 0.003426 seconds
Or...
Let's bring the rest of big-dataset.txt into memory...
$ vmtouch -vt big-dataset.txt
big-dataset.txt
[OOo oOOOOOOO] 6887/42116
[OOOOOOOOo oOOOOOOO] 10631/42116
[OOOOOOOOOOOOOOo oOOOOOOO] 15351/42116
[OOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 19719/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 24183/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 28615/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 31415/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 36775/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo oOOOOOOO] 39431/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 42116/42116
Files: 1
Directories: 0
Touched Pages: 42116 (164M)
Elapsed: 12.107 seconds