Is it possible to copy all files from one S3 bucket to another with s3cmd?

Solution 1:

s3cmd sync s3://from/this/bucket/ s3://to/this/bucket/

For available options, please use: $s3cmd --help

Solution 2:

AWS CLI seems to do the job perfectly, and has the bonus of being an officially supported tool.

aws s3 sync s3://mybucket s3://backup-mybucket

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Solution 3:

The answer with the most upvotes as I write this is this one:

s3cmd sync s3://from/this/bucket s3://to/this/bucket

It's a useful answer. But sometimes sync is not what you need (it deletes files, etc.). It took me a long time to figure out this non-scripting alternative to simply copy multiple files between buckets. (OK, in the case shown below it's not between buckets. It's between not-really-folders, but it works between buckets equally well.)

# Slightly verbose, slightly unintuitive, very useful:
s3cmd cp --recursive --exclude=* --include=file_prefix* s3://semarchy-inc/source1/ s3://semarchy-inc/target/

Explanation of the above command:

• –recursive
  In my mind, my requirement is not recursive. I simply want multiple files. But recursive in this context just tells s3cmd cp to handle multiple files. Great.
• –exclude
  It’s an odd way to think of the problem. Begin by recursively selecting all files. Next, exclude all files. Wait, what?
• –include
  Now we’re talking. Indicate the file prefix (or suffix or whatever pattern) that you want to include.
  s3://sourceBucket/ s3://targetBucket/
  This part is intuitive enough. Though technically it seems to violate the documented example from s3cmd help which indicates that a source object must be specified:
  s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]