Distributed file system with local disk cache

My server infrastructure is growing fast and I decided to create a distributed storage cluster. I've been looking for a proper filesystem for this task which meet my requirement, but none of them supports a local disk cache functionality. Each of my servers have a two 600GB SAS hard drives and I like to use them as a cache storage for most frequently accessed files from distributed storage.

Is any opensource filesystem supports this functionality? I like to use Ceph or GlusterFS, but I not found anything about local disk cache. I think it is a one of the basic features that distributed filesystem should support.

Check out OpenAFS it has local disk cache, see: http://docs.openafs.org/Reference/5/afs_cache.html

Another contender is XtreemFS: the feature set includes

In addition to full replicas that contain a complete copy, XtreemFS also supports partial replicas. These replicas are filled on demand when a client accesses data.

As per comments elsewhere - it would be possible to use local disk storage for caching gluster I/O albeit at the cost of VFS cache, AFS seems to be apropriate. But the big omissions from your question is whether you are trying to achieve fault-tolerance or performance, and whether the replicated storage should support transactions or frequent writes.

Other options include

• using a replicating nosql database
• bcache (which will provide performance improvements but not resillience improvements and poses problems with frequent writes / cache consistency)
• NAS/SAN