Faster s3 bucket duplication
I have been trying to find a better command line tool for duplicating buckets than s3cmd. s3cmd can duplicate buckets without having to download and upload each file. The command I normally run to duplicate buckets using s3cmd is:
s3cmd cp -r --acl-public s3://bucket1 s3://bucket2
This works, but it is very slow as it copies each file via the API one at a time. If s3cmd could run in parallel mode, I'd be very happy.
Are there other options available as a command line tools or code that people use to duplicate buckets that are faster than s3cmd?
Edit: Looks like s3cmd-modification is exactly what I'm looking for. Too bad it does not work. Are there any other options?
AWS CLI seems to do the job perfectly, and has the bonus of being an officially supported tool.
aws s3 sync s3://mybucket s3://backup-mybucket
http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
Supports concurrent transfers by default. See http://docs.aws.amazon.com/cli/latest/topic/s3-config.html#max-concurrent-requests
To quickly transfer a huge number of small files, run the script from an EC2 instance to decrease latency, and increase max_concurrent_requests to reduce the impact of latency. Eg:
aws configure set default.s3.max_concurrent_requests 200
If you don't mind using the AWS console, you can:
- Select all of the files/folders in the first bucket
- Click Actions > Copy
- Create a new bucket and select it
- Click Actions > Paste
It's still fairly slow, but you can leave it alone and let it do its thing.
I have tried cloning two buckets using the AWS web console, the s3cmd and the AWS CLI. Although these methods works most of the time, they are painfully slow.
Then I found s3s3mirror : a specialized tool for syncing two S3 buckets. It's multi-threaded and a lot faster than the other approaches I have tried. I quickly moved Giga-bytes of data from one AWS region to another.
Check it out at https://github.com/cobbzilla/s3s3mirror, or download a Docker container from https://registry.hub.docker.com/u/pmoust/s3s3mirror/