Upload only few bytes from a file to the SFTP

I have a use case wherein I need to upload only some bytes(can be in GB) of the file to SFTP server rather than the entire file. With SFTP PUT command the entire file gets uploaded, instead of some bytes.

I do not want to read the file and then make a smaller file out of that and then upload that entire smaller file to SFTP. This can be very memory intensive if the bytes to be uploaded are in GB. So, discarding this option.

Is there any other way to do this using SFTP(may be using -o to use ssh options) command. Perhaps something like scp: How to copy only the last 10% of a file?

pls suggest

Solution 1:

The standard OpenSSH sftp client doesn't support partial uploads.

If the server is running regular Linux, then you can directly pipe data to an SSH connection without using SFTP:

$ tail -c 2G ~/bigfile | ssh myserver "cat > smallchunk"

(Here tail -c 2G is just an example command that outputs the last 2 GB of the file. Replace it as appropriate – if you want the first X bytes then you'll want to use head, if you want want bytes from the middle use some combination of head|tail or maybe dd.)

Optional improvement: You can insert 'pv' to display a progress bar.

$ tail -c 2G ~/bigfile | pv | ssh myserver "cat > smallchunk"

If you do require SFTP, then the lftp client can be tricked into achieving this by uploading a named pipe:

1. Create a named pipe somewhere:

   $ mkfifo /tmp/pipe
   

2. Run a command that gets whichever bytes you want, and writes them to the named pipe:

   $ tail -c 2G ~/bigfile > /tmp/pipe &
   

   The command will immediately be paused as soon as it tries to write to the pipe. (This works exactly like the | nameless pipe in the previous example – the writing process gets paused until the reading process reads everything out of the pipe, 4kB at a time. Hence the & to let it happen in background.)

3. Ask lftp to upload the pipe:

   $ lftp sftp://myserver
   lftp> put -o smallchunk /tmp/pipe
   2G bytes transferred
   

For clarity regarding pipes: Here's example 1 again...

$ tail -c 2G ~/bigfile | ssh myserver "cat > smallchunk"

...and here's example 1 rewritten to use 'mkfifo' from example 2:

$ mkfifo /tmp/pipe
$ tail -c 2G ~/bigfile > /tmp/pipe &
$ ssh myserver "cat > smallchunk" < /tmp/pipe

Both of these work the same way: they use a pipe connecting one writer process to one reader. Both named and nameless pipes have a 4 kB buffer; every time it fills up, the writer is paused until the reader gets the data.