What's the fastest way to distribute a large file between a fleet of machines?
Imagine you run a highly parallel service which requires you to sync a large file (let's say 100GB) between many machines (let's say 100) on a regular basis, and you want to get this data distributed as quickly as possible, so the service which processes that file downstream can begin to do its work.
Sequentially sending it to machine one, then machine two, then machine three, etc etc, is easy to script, but very time consuming, and very wasteful: at any given time, 98 machines are sitting idle.
This wikipedia page describes a number of different algorithms for broadcasting data to multiple machines. Are there any shell utilities which use a smarter-than-naive strategy to perform this broadcast operation on a regular file?
If they are on the same network, any mutlicast FTP software should work (faster than bittorrent): uftp, udpcast, and flamethrower are available in apt.