Automatic file backup management?

I have a bunch of files (about 50 MB total) that I would like to back up every 20 minutes. Each backup should be compressed into a single (.tar.gz) file, and backups older than 24h should be removed. This process should be automated.

(I imagine this functionality is something that is widely sought for)

Short of writing my own application for this, what are my options?

Solution 1:

There are thousand of options. Since you have mentioned ubuntu, I have used the script from https://ubuntu.com/server/docs/backups-shell-scripts and modified it for you to delete archives older than 24h.

You can then run the script from a cron job every 20 minutes e.g.

*/20 * * * * /path/to/backup/script

#!/bin/bash                                                                                                                                                                                                                                   

# What to backup.                                                                                                                                                                                                                             
backup_files="/home /var/spool/mail /etc /root /boot /opt"

# Where to backup to.                                                                                                                                                                                                                         
dest="/mnt/backup"

# Create archive filename.                                                                                                                                                                                                                    
timestamp=$(date +%Y%m%d_%H%M%S)
hostname=$(hostname -s)
archive_file="$hostname-$timestamp.tgz"

# Backup the files using tar.                                                                                                                                                                                                                 
tar czf $dest/$archive_file $backup_files

# Cleanup backup archives older than 24h                                                                                                                                                                                                      
find $dest -name "*.tgz" -type f -mtime 1 -delete

Solution 2:

To generate a backup every 20 minutes, use cron. Your crontab could be

0,20,40 * * * *  cd /PATH; for file in *; do tar --backup=numbered -czf $file.tgz $file; done

The --backup option generates versioned backups like file.tgz.~1~, file.tgz.~2~ etc.

To remove files older than 24 hours, you could use another crontab entry, e.g.

30 * * * *      find /PATH -name '*.tgz*' -mtime +1 -exec rm -f {} +

This would run at each half hour, looking for files with tgz in their names and a modification time older than 24 hours (one day).