How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to be no tool to move whole bucket content to different bucket.

There is also problem on how to synchronize the data later on when we switch to EU bucket, the data that will be created meanwhile while the migration was running.

The new official AWS CLI natively supports most of the functionality of s3cmd.

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

aws s3 sync s3://oldbucket s3://newbucket --source-region us-west-1 --region us-west-2

I guess, you can run:

s3cmd --recursive cp s3://oldbucket/ s3://newbucket

It should copy it directly.

I don't know of a packaged solution to do this, but it is essentially just a bucket LIST and then PUT-copy each object from the US bucket to the EU bucket. The PUT-copy API supports "if-modified-since" and "if-none-match", so you can run it repeatedly without duplicating effort: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/index.html?RESTObjectCOPY.html

For a live migration, I would suggest the following approach:

1. Optional first step: If you have a huge amount of data to migrate, consider using AWS Import/Export to capture the first snapshot of the data on a physical device in the US and ship it physically to the EU. This may save you time and money.
   • Set up a script to migrate data from US to EU using the if-modified-since or if-none-match as described above. Make it smart enough not to clobber fresh EU data with stale US data (once you get to step 5 below). Run it continuously in a loop until the two buckets are in roughly the same state.
   • Modify your web app to PUT data to both the US and EU buckets at the same time, while continuing to GET data exclusively from the US.
   • Continue running your migration script until you are confident that the buckets are in exactly the same state (or as close as you can get, given eventual consistency).
   • Modify the web app again to GET data exclusively from the EU bucket.
   • You can now turn off your migration script.
   • Look for eventual consistency anomalies and clean them up.
   • Modify the web app to stop PUTting data to the US bucket.
   • DELETE everything in the US bucket.

You may want to use S3 Reduced Redundancy Storage on your EU bucket during the migration to get cheaper data rates and faster response times, since the data is just a duplicate of the US data.