RAM drive for compiling - is there such a thing?

An answer (see below) to one of the questions right here on Stack Overflow gave me an idea for a great little piece of software that could be invaluable to coders everywhere.

I'm imagining RAM drive software, but with one crucial difference - it would mirror a real folder on my hard drive. More specifically - the folder which contains the project I'm currently working on. This way any builds would be nearly instantaneous (or at least a couple orders of magnitude faster). The RAM drive would synchronize its contents with the hard disk drive in background using only idle resources.

A quick Google search revealed nothing, but perhaps I just don't know how to Google. Perhaps someone knows of such a software? Preferably free, but reasonable fees might be OK too.

Added: Some solutions have been suggested which I discarded in the very beginning. They would be (in no particular order):

• Buy a faster hard disk drive (SSD maybe or 10K RPM). I don't want a hardware solution. Not only software has the potential to be cheaper (freeware, anyone?), but it can also be used in environments where hardware modifications would be unwelcome if not impossible - say, at the office.
• Let OS/HDD do the caching - it knows better how to use your free RAM. The OS/HDD have generic cache algorithms that cache everything and try to predict which data will be most needed in the future. They have no idea that for me the priority is my project folder. And as we all know quite well - they don't really cache it much anyway. ;)
• There are plenty of RAM drives around; use one of those. Sorry, that would be reckless. I need my data to be synchronized back to the HDD whenever there is a bit of free time. In the case of a power failure I could bear losing the last five minutes of work, but not everything since my last checkin.

Added 2: An idea that came up - use a normal RAM drive plus a background folder synchronizer (but I do mean background). Is there any such thing?

Added 3: Interesting. I just tried out a simple RAM drive at work. The rebuild time drops from ~14 secs to ~7 secs (not bad), but incremental build is still at ~5 secs - just like on the HDD. Any ideas why? It uses aspnet_compiler and aspnet_merge. Perhaps they do something with other temp files elsewhere?

Added 4: Oh, nice new set of answers! :) OK, I've got a bit more info for all you naysayers. :)

One of the main reasons for this idea is not the above-mentioned software (14 secs build time), but another one that I didn't have access at the time. This other application has a 100 MB code base, and its full build takes about 5 minutes. Ah yes, it's in Delphi 5, so the compiler isn't too advanced. :) Putting the source on a RAM drive resulted in a BIG difference. I got it below a minute, I think. I haven't measured. So for all those who say that the OS can cache stuff better - I'd beg to differ.

Related Question:

RAM disk for speed up IDE

Note on first link: The question to which it links has been deleted because it was a duplicate. It asked:

What do you do while your code’s compiling?

And the answer by Dmitri Nesteruk to which I linked was:

I compile almost instantly. Partly due to my projects being small, partly due to the use of RAM disks.

In Linux (you never mentioned which OS you're on, so this could be relevant) you can create block devices from RAM and mount them like any other block device (that is, a HDD).

You can then create scripts that copy to and from that drive on start-up / shutdown, as well as periodically.

For example, you could set it up so you had ~/code and ~/code-real. Your RAM block gets mounted at ~/code on startup, and then everything from ~/code-real (which is on your standard hard drive) gets copied over. On shutdown everything would be copied (rsync'd would be faster) back from ~/code to ~/code-real. You would also probably want that script to run periodically, so you didn't lose much work in the event of a power failure, etc.

I don't do this anymore (I used it for Opera when the 9.5 beta was slow, no need anymore).

Here is how to create a RAM disk in Linux.

I'm surprised at how many people suggest that the OS can do a better job at figuring out your caching needs than you can in this specialized case. While I didn't do this for compiling, I did do it for similar processes and I ended up using a RAM disk with scripts that automated the synchronization.

In this case, I think I'd go with a modern source control system. At every compile it would check in the source code (along an experimental branch if needed) automatically so that every compile would result in the data being saved off.

To start development, start the RAM disk and pull the current base line. Do the editing, compile, edit, compile, etc. - all the while the edits are being saved for you.

Do the final check in when happy, and you don't even have to involve your regular hard disk drive.

But there are background synchronizers that will automate things - the issue is that they won't be optimized for programming either and may need to do full directory and file scans occasionally to catch changes. A source code control system is designed for exactly this purpose, so it would likely be lower overhead even though it exists in your build setup.

Keep in mind that a background sync task, in the case of a power outage, is undefined. You would end up having to figure out what was saved and what wasn't saved if things went wrong. With a defined save point (at each compile, or forced by hand) you'd have a pretty good idea that it was at least in a state where you thought you could compile it. Use a VCS and you can easily compare it to the previous code and see what changes you've applied already.

See Speeding up emerge with tmpfs (Gentoo Linux wiki).

Speeding up compiles using RAM drives under Gentoo was the subject of a how-to written many eons ago. It provides a concrete example of what has been done. The gist is that all source and build intermediate file are redirected to a RAM disk for compile, while final binaries are directed to the hard drive for install.

Also, I recommend exploring maintaining your source on hard drive, but git push your latest source changes to a clone respository that resides on the RAM disk. Compile the clone. Use your favorite script to copy the binaries created.

I hope that helps.

Use https://wiki.archlinux.org/index.php/Ramdisk to make the RAM disk.

Then I wrote these scripts to move directories to and from the RAM disk. Backup is made in a tar file before moving into the RAM disk. The benefit of doing it this way is that the path stays the same, so all your configuration files don't need to change. When you are done, use uramdir to bring back to disk.

Edit: Added C code that will run any command it is given on an interval in background. I am sending it tar with --update to update the archive if any changes.

I believe this general-purpose solution beats making a unique solution to something very simple. KISS

Make sure you change path to rdbackupd

ramdir

#!/bin/bash

# May need some error checking for bad input.

# Convert relative path to absolute
# /bin/pwd gets real path without symbolic link on my system and pwd
# keeps symbolic link. You may need to change it to suit your needs.
somedir=`cd $1; /bin/pwd`;
somedirparent=`dirname $somedir`

# Backup directory
/bin/tar cf $somedir.tar $somedir

# Copy, tried move like https://wiki.archlinux.org/index.php/Ramdisk
# suggests, but I got an error.
mkdir -p /mnt/ramdisk$somedir
/bin/cp -r  $somedir /mnt/ramdisk$somedirparent

# Remove  directory
/bin/rm -r $somedir

# Create symbolic link. It needs to be in parent of given folder.
/bin/ln -s /mnt/ramdisk$somedir $somedirparent

#Run updater
~/bin/rdbackupd "/bin/tar -uf $somedir.tar $somedir" &

uramdir

#!/bin/bash

#Convert relative path to absolute
#somepath would probably make more sense
# pwd and not /bin/pwd so we get a symbolic path.
somedir=`cd $1; pwd`;

# Remove symbolic link
rm $somedir

# Copy dir back
/bin/cp -r /mnt/ramdisk$somedir $somedir

# Remove from ramdisk
/bin/rm -r /mnt/ramdisk$somedir

# Stop
killall rdbackupd

rdbackupd.cpp

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <signal.h>
#include <sys/time.h>

struct itimerval it;
char* command;

void update_archive(int sig)
{
    system(command);
}

int main(int argc, char**argv)
{
    it.it_value.tv_sec     = 1;   // Start right now
    it.it_value.tv_usec    = 0;
    it.it_interval.tv_sec  = 60;  // Run every 60 seconds
    it.it_interval.tv_usec = 0;

    if (argc < 2)
    {
        printf("rdbackupd: Need command to run\n");
        return 1;
    }
    command = argv[1];

    signal(SIGALRM, update_archive);
    setitimer(ITIMER_REAL, &it, NULL); // Start

    while(true);

    return 0;
}

We used to do this years ago for a 4GL macro-compiler; if you put the macro library and support libraries and your code on a RAM disk, compiling an application (on an 80286) would go from 20 minutes to 30 seconds.