Share /home between datacenters
I have two servers, placed at datacenters in holland and france. Both are running Debian Wheezy. I need to share /home between them, with good performance. There are 300-something users on the servers, around 30 of them should be able to have active processes on a given server at a given time, each having 50 kbit reads and and 20 kbit writes/second, with short peaks around 2000 kbit/s reading. measures with iotop on local storage. I have a lot of small files, around 500000 in total and need as low latency as possible. Ping between servers are 17 ms, and the connection is able to reach around 20-30 MB/s when using scp and wget. It seems there should be plenty of bandwidth available for it too work, but...
What I've trued so far: sshfs: Seemed like it had better performance than nfs, but it ranomly changed permissions of files to root, making the application crash.
nfs: Way to slow, tried noatime an a bunch of other options, but it keeps acting sluggish, even when only a few processes are active.
drbd: 5 hours of dead-end work, when I realized I couldn't actually mount the filesystem on both systems :-(
glusterfs: Having a local copy of all data really sounded promising, but random file access is really slow and after running a while, it becomes unbelievable slow and almost hangs. noatime doesn't help.
nfs again: Still sluggish.
Crying into the keyboard: No improvement at all.
What to try next? Each of the failed trials have taken an evening or maybe more during the last week, and I'd really like the next method to work. And yes, it crucial that the filesystems are shared between both servers.
Thanks for any new ideas on this problem.
Solution 1:
There are some possible solutions for this:
You can go for a replicated block storage like DRBD (or MARS as mentioned above), but you need to setup a cluster file system on top of the block storage. Such file systems could be GFS2 or OCFS2 which are both available in the Debian kernel afaik. DRBD can handle primary/primary and you can mount it on both servers at the same time. But if you do this with a standard file system, one server does not know about the other and you would destroy your file system in a few seconds. A cluster file system on top would handle the communication and locking so that both nodes can write to the same block.
Use a distributed file system for /home. You will find a list of such file systems at http://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems. But beware and choose wisely. They all can't do magic and all have their drawbacks. Gluster is such a file system. For some systems, you might need more than only two nodes.
If it does not have to be replicated in real time and a nearly-realtime file sync would be sufficient, then have a look at BitTorrent Sync (http://www.getsync.com/), Dropbox or alternatives. Each server has it's own /home, but changes get replicated on file basis to the other server.