Extract duplicity files manually
I lost the drive, and the Ubuntu OS installed in it, from which I created a backup using Duplicity, and I don't know how to restore the archived files I still keep in another drive. They all follow these 4 notations
duplicity-full.20140106T111233Z.manifest
duplicity-full-signatures.20140106T111233Z.sigtar.gz
duplicity-full.20140106T111233Z.volxxx.difftar.gz (multiple volumes)
duplicity-full.20140106T111233Z.volxxx.difftar (multiple volumes)
There's important data I need to recover. Is there anything I can do? Would be enough to use gzip? And if so, how could I extract them all easily?
EDIT 1 Trying your suggestions, both with the restore and the --gio options, I get this text:
duplicity restore file://media/ubuntu/Toshiba\ HDD/BACKUPS/Documents/ /media/ubuntu/9fb63c8e-ecb5-4c55-b4e0-282e7b4a82ff/tmp/docs/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase:
Traceback (most recent call last):
File "/usr/bin/duplicity", line 1494, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1488, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1337, in main
do_backup(action)
File "/usr/bin/duplicity", line 1422, in do_backup
restore(col_stats)
File "/usr/bin/duplicity", line 697, in restore
restore_get_patched_rop_iter(col_stats)):
File "/usr/bin/duplicity", line 719, in restore_get_patched_rop_iter
backup_chain = col_stats.get_backup_chain_at_time(time)
File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 952, in get_backup_chain_at_time
raise CollectionsError("No backup chains found")
CollectionsError: No backup chains found
/media/ubuntu/Toshiba\ HDD/BACKUPS/Documents/ is where all the duplicity files are, and /media/ubuntu/9fb63c8e-ecb5-4c55-b4e0-282e7b4a82ff/tmp/docs/ a temporary destination folder I want to restore the files to.
EDIT 2 Corrected the missing slash in file://media.... Now I get restored files, but not exactly what I expected. There are a lot of files missing (up to 24 GB in there!)
root@ubuntu:/media/ubuntu/Toshiba HDD/BACKUPS/Documents# ls -la /tmp/docs/home/luis/
total 4176
drwx------ 16 root root 1000 Jun 30 19:13 .
drwx------ 3 root root 60 Jun 30 19:13 ..
drwx------ 3 1000 1000 60 Oct 23 2013 .activestate
drwxr-xr-x 2 1000 1000 60 Oct 23 2013 .ActiveState
drwx------ 3 1000 1000 60 Dec 28 2012 .adobe
-rw------- 1 1000 1000 32120 Jan 4 12:56 .bash_history
-rw-r--r-- 1 1000 1000 220 Dec 28 2012 .bash_logout
-rw-r--r-- 1 1000 1000 3637 Dec 28 2012 .bashrc
drwx------ 3 1000 1000 60 Jan 3 14:01 .cache
drwxr-xr-x 3 1000 1000 60 Nov 16 2013 .cddb
-rw-rw-r-- 1 1000 1000 740 Jan 10 2013 .cola
drwx------ 3 1000 1000 60 Dec 29 2012 .compiz
drwxrwxr-x 3 1000 1000 80 Oct 1 2013 .composer
drwx------ 46 1000 1000 1020 Dec 8 2013 .config
drwx------ 3 1000 1000 60 Dec 28 2012 .dbus
-rw-r--r-- 1 1000 1000 25 Jul 13 2013 .dmrc
drwx------ 3 1000 1000 380 Jan 6 10:31 .dropbox
drwxr-xr-x 6 1000 1000 1660 Dec 17 2013 .dropbox-dist
drwx------ 3 root root 60 Jun 30 19:13 .eclipse
-rw------- 1 1000 1000 50282 Jan 6 10:29 .ICEauthority
drwxrwxr-x 14 1000 1000 340 Dec 7 2013 .PlayOnLinux
-rw-r--r-- 1 root root 2077491 Nov 23 2013 .Soulseek.1385165090491
-rw-r--r-- 1 root root 2076644 Nov 23 2013 .Soulseek.1385166430938
-rw-r--r-- 1 1000 1000 9986 Dec 30 20:02 .Soulseek.1388433748295
drwxrwxr-x 2 1000 1000 60 Oct 13 2013 .SyncWall
-rw------- 1 1000 1000 109 Jan 6 10:29 .Xauthority
-rw-rw-r-- 1 1000 1000 0 Jul 29 2013 .Xauthority.25Y20W
-rw-rw-r-- 1 1000 1000 0 Jul 13 2013 .Xauthority.7K14ZW
-rw-rw-r-- 1 1000 1000 0 Jul 7 2013 .Xauthority.7K7SZW
-rw-rw-r-- 1 1000 1000 0 Jul 13 2013 .Xauthority.9X1E0W
-rw-rw-r-- 1 1000 1000 0 Sep 1 2013 .Xauthority.A3D52W
-rw-rw-r-- 1 1000 1000 0 Sep 1 2013 .Xauthority.CEUV2W
-rw-rw-r-- 1 1000 1000 0 Jul 27 2013 .Xauthority.CP7Q0W
-rw-rw-r-- 1 1000 1000 0 Sep 8 2013 .Xauthority.E1ET2W
-rw-rw-r-- 1 1000 1000 0 Aug 26 2013 .Xauthority.E32K2W
-rw-rw-r-- 1 1000 1000 0 Sep 14 2013 .Xauthority.EKK92W
-rw-rw-r-- 1 1000 1000 0 Jul 12 2013 .Xauthority.F4QRZW
-rw-rw-r-- 1 1000 1000 0 Jul 25 2013 .Xauthority.ISVZ0W
-rw-rw-r-- 1 1000 1000 0 Jul 11 2013 .Xauthority.JU3UZW
-rw-rw-r-- 1 1000 1000 0 Aug 31 2013 .Xauthority.M30S2W
-rw-rw-r-- 1 1000 1000 0 Aug 1 2013 .Xauthority.M3H20W
-rw-rw-r-- 1 1000 1000 0 Sep 9 2013 .Xauthority.MKIN2W
-rw-rw-r-- 1 1000 1000 0 Aug 28 2013 .Xauthority.RWHB2W
-rw-rw-r-- 1 1000 1000 0 Jul 15 2013 .Xauthority.SN85ZW
-rw-rw-r-- 1 1000 1000 0 Sep 15 2013 .Xauthority.T22C3W
-rw-rw-r-- 1 1000 1000 0 Jul 14 2013 .Xauthority.T6CB0W
-rw-rw-r-- 1 1000 1000 0 Jul 31 2013 .Xauthority.VF7Q0W
-rw-rw-r-- 1 1000 1000 0 Sep 9 2013 .Xauthority.WGEX2W
-rw-rw-r-- 1 1000 1000 0 Aug 31 2013 .Xauthority.WZR52W
-rw-rw-r-- 1 1000 1000 0 Jul 25 2013 .Xauthority.YQIR0W
Most strange is, besides all the files starting with a dot, that I can't browse the home/luis directories with Nautilus, which in fact were where all my backed up data was.
-
Use
duplicity
to extract it:duplicity restore file:///path_to_folder_contains_backups/ path_where_to_extract_it/
Enter GnuPG passphrase, Press Enter
By default, Duplicity restores last backup (comparing time) in the source folder even its not the backup for the same folder. If you have multiple backups for different folders or you want specific time of same folder, use --time
option example:
duplicity restore --time 20140106T111233Z file:///path_to_folder_contains_backups/ path_where_to_extract_it/
Unable to see the files you are looking for, try:
-
Create function that list all files contained in all full/inc backups available in the source folder:
ldup () { for f in $(ls $1/duplicity-*.manifest.gpg); do echo "========================================================================"; echo $f; n=$(basename $f .manifest.gpg); prefix=$(echo $n | awk 'BEGIN { FS = "." } ;{ print $1 }'); if [[ "$prefix" == "duplicity-full" ]]; then t=$(echo $n | awk 'BEGIN { FS = "." } ;{ print $2 }'); else t=$(echo $n | awk 'BEGIN { FS = "." } ;{ print $4 }'); fi; echo -e "Creation time:\t" $t; echo "------------------------------------------------------------------------"; duplicity list-current-files --time $t file://$1/ | grep "$2"; done; }
-
To list all files (note without tail
/
):ldup /path_to_folder_contains_backups
-
search for a pattern in file names:
ldup /path_to_folder_contains_backups pattern
Reference: man duplicity
First try to restore with duplicity.duplicity --gio file:///media/backup /tmp/restore
There are some reasonably detailed (and mostly correct) steps over here.
Key details
The key point is to unpack all of the duplicity-full.*.difftar.gz
files in the same place, so that you're left with just two snapshot/
and multivol_snapshot/
directories.
If your file is in snapshot/
then you're done. Otherwise find the directory in multivol_snapshot/
at the path where your file used to be: you need to join together all the files in this directory to recreate the original file. The files are numbered, and can be joined together using the cat
command. Depending on how large the original was, there may be many parts.
Problem with original instructions
The directions linked above suggest using cat * > rescued-file
. Unfortunately this simple approach fails if you have more than 9 parts. Since *
expands in dictionary order, not numeric order, 10
would be listed before 2
, and the file would be reconstructed in the wrong order.
Workaround
One simple approach is to remember that dictionary order does work when numbers are the same length, and that ?
matches a single character. So if your largest file has three digits, you can manually enter:
cat ? ?? ??? > rescued-file
Add or remove ?
patterns as necessary, depending on the largest file number.
Script
If you have a lot of files to recover and don't fancy typing that for all of them, you might prefer to use a script such as this. It lists the containing directory for every file, removes duplicates from the list, then goes to each directory and creates a content
file from the fragments there. (spacer
is just to make $1
work.)
find multivol_snapshot/ -type f -printf '%h\0' | \
sort -uz | \
xargs -0 -n 1 sh -c 'cd "$1" ; cat $(ls | sort -n) > content' spacer
Now you just have to add /content
to the end of any filename you were looking for, and you should find it.
Limitations
This doesn't restore any of the original file permissions or ownership. It also doesn't deal with incremental backups, but then the linked instructions also hit a bit of a dead end on this point — they just suggest using rdiff
'to stitch the files together' and refer the reader to man rdiff
.
source : SF SE
here you can find a java program for easy restoration of complicated backup.