how to specify destination for s3 bucket download python

how to specify destination path for the downloaded file?

    s3 = boto3.resource(..)
    my_bucket = s3.Bucket(S3_BUCKET)
    # download file into current directory
    for s3_object in my_bucket.objects.all():
        path, filename = os.path.split(s3_object.key)
        my_bucket.download_file(s3_object.key, filename)

Solution 1:

In this line:

my_bucket.download_file(s3_object.key, filename)

the filename parameter specifies where to save the object on the local disk.

The Key of an object in Amazon S3 can include a path, such as january/invoice.txt. However, this line in your code:

path, filename = os.path.split(s3_object.key)

is splitting-out the path from the filename, leaving path = january/ and filename = invoice.txt.

Therefore, when your code is saving objects, it will be saving them in the current directory.

To modify this, you should specify a full path in the filename, such as:

my_bucket.download_file(s3_object.key, '/tmp/' + filename)

Things get a bit more complex when you want to preserve the directory structure. For example, there might be multiple files with the same name:

january/invoice.txt
february/invoice.txt

The existing code would overwrite the first invoice.txt with the second one. If you wish to preserve the directory structure, your code would not call os.path.split but would instead need to ensure that each destination directory already exists.

For an example, see: Download a folder from S3 using Boto3