move all files in local directory to remote directory
I want to move all files in local directory to remote directory.
And I am using this code to achieve it
SOURCE_FILE=/var/www/oneserver/*
TARGET_DIR=/var/www/anotherServer
ARCHIEVEFILE=/var/www/archieveServer
/usr/bin/expect<<EOD
spawn /usr/bin/sftp -o Port=$PORT $USER@$HOST
expect "password:"
send "$PASSWORD\r"
expect "sftp>"
send "put $SOURCE_FILE $TARGET_DIR\r"
expect "sftp>"
send "bye\r"
EOD
It works fine, but sometime it stops and only send some files.
I also want to move the already sent file to ARCHIEVEFILE by using mv command. But dont get idea how to move it. I can't use scp
because the remote server not allowed using basic port, and the only way is using sftp.
Can anybody help, please?
[EDIT]
the $ARCHIEVEFILE
is still the local server, just to backup/move so those files are not sent anymore with next cronjob
the TARGET_DIR
is the remote server.
There's no command in OpenSSH sftp
to move files to remote directory.
What you can do is to:
- use
sftp
put
to upload the files (as you are doing already), and then -
use shell
rm
command to delete the files aftersftp
is done (i.e. afterEOD
):rm $SOURCE_FILE
Or use
!
to escape to shell fromsftp
script, after you sentput
:send "!rm $SOURCE_FILE\r" expect "sftp>"
Of course, this is not an atomic solution. If a file is added between put
and rm
, it will be lost. For an atomic solution, you have to iterate files in a local directory and upload and delete them one by one. Also for a robust solution, you need to check if an upload was successful.
The following approach may simplify things a lot:
- Use
sshfs
to mount the remote share(s) as local path(s). - Work with
cp
and/ormv
as if all operations were local.
Compare this answer of mine.
You still need some logic to detect if cp
to the remote location succeeded, only then mv
to the local archive; otherwise retry or something. But now all SFTP-related work should be transparently handled by sshfs
.
Additionally shell globbing and quotes you use makes your code prone to errors related to spaces in filenames etc.
After you mount the remote share e.g. in /mnt/a b/remote
, it will be so much easier to handle this.
(Note: it is a good practice to use lowercase variable names).
-
To mount:
sshfs -p $port $user@$host:"/path/on/the/remote/host/" "/mnt/a b/remote/"
Use key based authentication or read Username and password in command line with sshfs. Read about security concerns.
-
A code stub to copy files:
# I deliberately use paths with spaces to show how to handle them for filename in "/source/location with spaces"/*; do cp "$filename" "/mnt/a b/remote/" && mv "$filename" "/archive/location with spaces/" done
This
&&
ensuresmv
will run only ifcp
succeeds. -
To unmount:
fusermount -u "/mnt/a b/remote/"