Move data from Glacier vault to Glacier Deep Archive

Solution 1:

I looked at this when Glacier came out, and I posted a comment on the AWS blog that was never replied to.

Best I can tell there is no migration path from Glacier to Glacier Cold Archive. You will have to migrate the data manually.

I have two suggested approaches:

Local Upload

If you have the data locally and are confident of its integrity simply use the AWS command line or any tool you desire to upload it. You may want to tweak the S3 parameters in your config file to speed this, which can increase your internet bandwidth utilization by using more threads. This is especially useful if you have a lot of small files, with large files you could potentially max out your bandwidth.

Download then Upload

Second approach is to

• Restore the data from Glacier
• Download the data to a computer, either local or ideally an EC2 on demand instance (not spot as you may lose your data if your instance is terminated)
• Upload the data to S3 using the IA tier

Create a User

Here's the S3 command I use for upload from Windows. Note that you need a profile "glacier writer"

You'll have created an IAM user that has access to that bucket, and any other resources you need. Have their access / secret keys available. If you need to do this with a role it's a bit more work but not difficult, there's docs online

aws configure --glacier-writer

You can then edit your configure file to include this or similar. This works well on my home internet connection as I have 20Mbps upload. If you have high bandwidth and a fast machine you can increase the concurrent requests. I've successfully used up to 80 threads on high bandwidth corporate connections, which takes 1-2 xeon cores.

[profile glacier-writer]
region = us-west-2
output = json
s3 =
    max_concurrent_requests = 10
    max_queue_size = 100
    multipart_chunksize = 75MB
    multipart_threshold = 200MB

On Windows this is in

c:\users\username\.aws\configure

On Linux it's in

~\home\.aws\configure

Do the Upload

A simple S3 sync is what I do, but you can also use "s3 cp" to simply upload to S3.

aws s3 sync C:\Source\Folder\ s3://bucket-name/ --profile glacier-writer --storage-class DEEP_ARCHIVE --exclude *.tmp

Solution 2:

You can now use Amazon S3 Glacier Re:Freezer. https://aws.amazon.com/about-aws/whats-new/2021/04/new-aws-solutions-implementation-amazon-s3-glacier-re-freezer/

It is a serverless solution that automatically copies entire Amazon S3 Glacier vault archives to a defined destination Amazon Simple Storage Service (Amazon S3) bucket and S3 storage class. The solution automates the optimized restore, copy, and transfer process and provides a prebuilt Amazon CloudWatch dashboard to visualize the copy operation progress. Deploying this solution allows you to seamlessly copy your S3 Glacier vault archives to more cost effective storage locations such as the Amazon S3 Glacier Deep Archive storage class.