send files to dozens of servers quickly

This was an interview question once for me...

Multicast options:
BitTorrent
Other protocols using pub/sub messaging.

Another approach... Use a distribution tree:
Send to N hosts, who will in turn each send to N hosts; and work down the tree that way. That would of course require some development work on your side, but it's possibly the more scalable approach.

Most of this depends on how many systems you'll actually need to cater to, the index size and your networking infrastructure.


You might be better served by using a shared filesystem, especially if the search index is readonly by the app using the search index (i.e. at the destination end). That way, much of the complexity is done for you.