RAM disk and physical RAID
I need to create a lot of temporary files for which I need to have fast access, so I'm thinking about using a RAM disk. The problem is that the temporary files might get pretty big (1-4 GB) and in this case the disk-storage would be OK.
Is there any way to create a mix between a RAM disk and a physical HDD, so that the HDD gets used in the worst case (when I meet the big temporary files)?
---Edit---
I cannot afford faster SSD's, and I don't need persistent storage, and I don't need 100 GB of temporary data. I'm using my home computer for this (2 GHz AMD system...). I received a tip about UnionFS. What is the experience with it?
Solution 1:
Use tmpfs and a big swap partition or file. This file system will cache data in memory as long as it can, and swap them to disk if they don't fit into RAM.
Solution 2:
Why not try using the solid state SSD drives? You can get those with a capacity of 100 GB or more, and they are way faster than standard disks.
Solution 3:
I'd second the option to do RAID 0 (stripe) with a hardware RAID controller. I quoted a video encoding server and the scratch drives are 15k SAS in RAID 0 and it chews through video like nothing else.