How to transfer large files between two remotes by using a third machine?

I need to automatically mirror a large amount (terabytes) of files in two unix machines over a slow link (1 Mbps). This needs to be done frequently, but the data doesn't change too much (delta transmission doesn't saturate the link).

The usual solution would be rsync, but there's an additional requirement: it's undesirable, from a security standpoint, that either the source or destination machines have (keyless) ssh keys to each other, or any kind of filesystem access. All communication between the two machines should thus be initialized (and mediated) through a third machine.

I've asked a separate question about rsync in particular here. Are there other obvious solutions I'm missing?

Not exactly what you are asking. But just in case...

You can easily authorize a specific ssh key to only be allowed to run certain commands on the remote machine (this is similar to how gitosis and gitolite and possibly any other secure git-over-ssh implementations work). Long story short, add these options to the destination users's authorized_keys file:

• command="/path/to/validation/script"
• no-port-forwarding
• no-X11-forwarding
• no-agent-forwarding
• no-pty

The file should look like this in the end:

command="/path/to/validation/script",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2E.....

The no-* options are self explanatory.

The command= option runs a "forced" command "ignoring any command supplied by the client". The command originally supplied by the client is available in the SSH_ORIGINAL_COMMAND environment variable" (somewhat quoted from the sshd man page). In other words when someone logs in with that key, the only command that will be run is the validation script. It is up to you to allow them to run it or not. So, your validation script can do something like:

#!/bin/bash
if [[ "$SSH_ORIGINAL_COMMAND" == "rsync --server -vtr --delete /source/path/ /destination/path/" ]]; then
     $SSH_ORIGINAL_COMMAND
fi

And due to the other no- options, this key is not allowed to do anything else too harmful.

Update: There is a ready-made script that simplifies this configuration a little. -- Mikko

I think your biggest need is to transfer only the delta and so if you don't want the security compromise/overhead I suspect you'll have to keep a third copy to do that.

I've been doing this (no delta!):

dir=`mktemp -d` && cd $dir \
&& rsync -avz user1@host1:~/source . \
&& rsync -avz . user2@host2:~/dest \
&& rm -rvf $dir

You would need

rsync -avz user1@host1:~/source mirror \
&& rsync -avz mirror user2@host2:~/dest \

And possibly want --delete too