multipart upload to S3 with Django-storages

I'm using django-storages to upload large files into s3. Currently testing with files that are 1GB in size and would like to split it into multi part for quicker uploads.

I have tried setting AWS_S3_MAX_MEMORY_SIZE to 5mb to see if that would help, but no luck. I could not find any other settings for multipart uploads in the django-storage docs https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html.

Solution 1:

I've found this issue: https://github.com/jschneier/django-storages/issues/30

It seems the support for multipart-upload was added not long after that. Since then the code has changed a bit, but here you can find the most recent code: https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py

See the __init-method of the S3Boto3StorageFile class: you can specify a buffer_size as parameter (min is 5MB - requirement of AWS), or you specify this via setting AWS_S3_FILE_BUFFER_SIZE .

I see that this is not in the documentation, but the code suggests that it should work. Hope this helps (haven't tried it myself yet).

Edit: found this overview of available settings: https://gist.github.com/datagrok/2414f981380c751538751e7b3ab8e3e8