Live file syncronization across multiple Linux servers with millions of files and directories
What is the best way to synchronize huge data of a running production server?
Our server has over 20 million files (small files with 10k and bigger files up to 50MB) stored in 1 millon directories. The size of all data is about 5 TB (steadily increasing).
Is it possible to synchronize the data with lsyncd and what are the limits (especially of inotify)? How much additional space does lsyncd need? What about the load (cpu and memory) and the live time?
Another solution would be GlusterFS. Is it possible to use GlusterFS on a production with none or minimal downtime? GlusterFS stores a lot of magic data in x-attributes files and the storage volume is about 15 to 20% bigger than systems with non GlusterFS. Seems like a huge amount of waste...? What about the load?
And at least rsync and cronjobs could do the job. rsync would only run on the slave... So no additional space is needed on the primary server, but rsync must read the full directory tree every time the cron runs...
Solution 1:
I'd seriously recommend using something like ZFS for the filesystem.
Built-in tools like the ZFS snapshot and ZFS send/receive allow you to take block-level snaps of the filesystem and ship it to a second server.
Some third party tools like sanoid/syncoid can set automatic management/pruning and synchronization of your filesystem from one host to another.
This is done at the block device level, so you avoid the rsync checksum/inventory process.
Solution 2:
If you cannot change the filesystem on the production server, I would put the files on another server and mount them with NFS. I would use Linux and ZFS if man-hours are inexpensive, maybe some kind of home NAS distribution or maybe even a home NAS (both probably ZFS-based) if everything is expensive and you can find one that does professional-level redundancy, or a NetApp or an IBM Spectrum Scale if money is not a problem compared to reliability and support.
Once you have the files on a real full-featured file server with professional-level redundancy, you point your backup server either directly to the primary NFS IP if you configured failover, or to the backup NFS server.