Downloading an entire S3 bucket?

Solution 1:

AWS CLI

See the "AWS CLI Command Reference" for more information.

AWS recently released their Command Line Tools, which work much like boto and can be installed using

sudo easy_install awscli

or

sudo pip install awscli

Once installed, you can then simply run:

aws s3 sync s3://<source_bucket> <local_destination>

For example:

aws s3 sync s3://mybucket .

will download all the objects in mybucket to the current directory.

And will output:

download: s3://mybucket/test.txt to test.txt
download: s3://mybucket/test2.txt to test2.txt

This will download all of your files using a one-way sync. It will not delete any existing files in your current directory unless you specify --delete, and it won't change or delete any files on S3.

You can also do S3 bucket to S3 bucket, or local to S3 bucket sync.

Check out the documentation and other examples.

Whereas the above example is how to download a full bucket, you can also download a folder recursively by performing

aws s3 cp s3://BUCKETNAME/PATH/TO/FOLDER LocalFolderName --recursive

This will instruct the CLI to download all files and folder keys recursively within the PATH/TO/FOLDER directory within the BUCKETNAME bucket.

Solution 2:

You can use s3cmd to download your bucket:

s3cmd --configure
s3cmd sync s3://bucketnamehere/folder /destination/folder

There is another tool you can use called rclone. This is a code sample in the Rclone documentation:

rclone sync /home/local/directory remote:bucket

Solution 3:

I've used a few different methods to copy Amazon S3 data to a local machine, including s3cmd, and by far the easiest is Cyberduck.

All you need to do is enter your Amazon credentials and use the simple interface to download, upload, sync any of your buckets, folders or files.

Solution 4:

You've many options to do that, but the best one is using the AWS CLI.

Here's a walk-through:

1. Download and install AWS CLI in your machine:

   • Install the AWS CLI using the MSI Installer (Windows).
   • Install the AWS CLI using the Bundled Installer for Linux, OS X, or Unix.

2. Configure AWS CLI:

   

   Make sure you input valid access and secret keys, which you received when you created the account.

3. Sync the S3 bucket using:

   aws s3 sync s3://yourbucket /local/path
   

   In the above command, replace the following fields:

   • yourbucket >> your S3 bucket that you want to download.
   • /local/path >> path in your local system where you want to download all the files.