How to make a local tar file of a remote directory

This link describes how to copy tarred files to minimise the amount of date sent over the network. I am trying to do something slightly different.

I have a number of remote files on different subdirectory levels:

remote:/directory/subdir1/file1.ext
remote:/directory/subdir1/subsubdir11/file11.ext
remote:/directory/subdir2/subsubdir21/file21.ext

And I have a file that lists all of them:

remote:/directory/allfiles.txt

To copy them most efficiently, on the remote site I could just do

tar zcvf allfiles.tgz `cat /directory/allfiles.txt`

but there is not enough space to do that.

I do have enough storage space on my local disc. Is there a way to tar an incoming stream from a remote server (using scp or ssh for the transfer)?

Something like /localdir$ tar zc - | ssh remote `cat /directory/allfiles.txt` I would guess - but that would only list the remote files on the local host.

Solution 1:

You got it almost right, just run the tar on the remote host instead of locally. The command should look something like the following:

ssh remote_host tar cvfz - -T /directory/allfiles.txt > remote_files.tar.gz

Solution 2:

If you have enough space on your local disk and your goal is to minimise the amount of date sent over the network maybe it is enough to enable the compression with scp or rsync :

scp -avrC remotehost:/path/to/files/file1 /files/file2 ...  local/destination/path

Of course you can do a little script to loop over each file and do an scp compressed transfer, even without the use of tar. All is more cosy with rsync

rsync -avz --files-from=FILE remotehost:/path/to/files  local/destination/path

You can connect via ssh to the remote host and write there

tar cvzf - -T list_of_filenames | ssh Local_Hostname tar xzf -

References:

• from man scp:

  -C      Compression enable.  Passes the -C flag to ssh(1) to enable compression.
  

• from man rsync

  --files-from=FILE       read list of source-file names from FILE
  -z, --compress          compress file data during the transfer
  --compress-level=NUM    explicitly set compression level