Backup AWS EFS to S3
I've been desperately trying to find a way to backup my AWS EFS file system to S3, but cannot seem to find one.
There's several EC2 instances running all having access to the mentioned EFS. In order to reduce traffic, I already tried launching a Lambda Function, which SSHs to the EFS instances and runs "aws s3 sync ...". Unfortunately SSHing from Lambda services doesn't seem like a good production ready solution.
What I've also tried was adapting DataPipeline, but launching additional instances just for backups seems like a hassle, too.
Isn't there some easy way of backing up EFS to S3?
Any suggestions appreciated.
Actually, I think S3 Sync is what you want. Maybe setup Cron on the EC2 instances and invoke S3 Sync that way? Are you using ECS as well? I have a Cron container that does the job pretty well. For those reading who are not familiar with AWS CLI (https://aws.amazon.com/cli/) the syntax for S3 Sync is like:
aws s3 sync /path/to/source/ s3://bucket/destination/
- Back up EFS using a tool such as Attic to create a compressed, incremental, de-duplicated backup on one EC2 instance.
- Use S3FS or the S3 API to upload those files to S3. Personally I use a dropbox upload script, which works fine as well.
Note that Attic runs at whatever interval you specify, but keeps only the checkpoints you specify. For example you might have daily backups, but then it only keeps monthly after the first month, and yearly after the first year. Because of this it deletes files from storage. If you don't delete the files from your repository it won't hurt, but you will use more storage than required. That's why a sync of the Attic backup files might be better than a copy.