Copy directory structure intact to AWS S3 bucket
I want to use the AWS S3 cli to copy a full directory structure to an S3 bucket.
So far, everything I've tried copies the files to the bucket, but the directory structure is collapsed. (to say it another way, each file is copied into the root directory of the bucket)
The command I use is:
aws s3 cp --recursive ./logdata/ s3://bucketname/
I've also tried leaving off the trailing slash on my source designation (ie, the copy from argument). I've also used a wildcard to designate all files ... each thing I try simply copies the log files into the root directory of the bucket.
(Improving the solution of Shishir)
- Save the following script in a file (I named the file
s3Copy.sh
)
path=$1 # the path of the directory where the files and directories that need to be copied are located
s3Dir=$2 # the s3 bucket path
for entry in "$path"/*; do
name=`echo $entry | sed 's/.*\///'` # getting the name of the file or directory
if [[ -d $entry ]]; then # if it is a directory
aws s3 cp --recursive "$name" "$s3Dir/$name/"
else # if it is a file
aws s3 cp "$name" "$s3Dir/"
fi
done
- Run it as follows:
/PATH/TO/s3Copy.sh /PATH/TO/ROOT/DIR/OF/SOURCE/FILESandDIRS PATH/OF/S3/BUCKET
For example ifs3Copy.sh
is stored in the home directory and I want to copy all the files and directories located in the current directory, then I run this:~/s3Copy.sh . s3://XXX/myBucket
You can easily modify the script to allow for other arguments of s3 cp
such as --include
, --exclude
, ...
I believe sync is the method you want. Try this instead:
aws s3 sync ./logdata s3://bucketname/