Can I ignore files by pattern in deja-dup (Backup)?

My Déjà Dup backups have become quiet large and I noticed they contain a huge number of unnecessary files (e.g. *.pyc files, **__pycache__ folders and other build related temporary stuff).

I know that I can ignore specific folders, but is there a way to exclude files and or folders by patterns?

I thought there might be more options usable trough a configuration file, but Déjà Dup doesn't use one. So I looked at duplicity (the CLI it is based on), but the man page doesn't mention a configuration file either. I know that duplicity can ignore files and folders based on patterns (--exclude, --exclude-filelist), but I have no idea how to combine this with Déjà Dup.

Do I have to ditch Déjà Dup and use duplicity manually? Or is there a way to set the needed options, so that they are used automatically, when duplicity is used by Déjà Dup?

You can edit the exclude list like:

gsettings get org.gnome.DejaDup exclude-list
# remove comment to execute
# gsettings set org.gnome.DejaDup exclude-list ['path1', 'path2']

Source: https://answers.launchpad.net/deja-dup/+question/280954

I tried to add patterns like '**/.git' and '**/build' into that list, like this:

gsettings get org.gnome.DejaDup exclude-list > exclude-list
gedit exclude-list
gsettings set org.gnome.DejaDup exclude-list "`cat exclude-list`"

But to me it seems like the **'s were not passed to duplicity. So instead I ended up doing seaches like

locate "/home/*/.svn"
locate "/home/*/build"

and added them to the exclude-list manually

There is no way currently with Deja Dup to do advanced filtering like that. See upstream bug https://bugs.launchpad.net/deja-dup/+bug/374274

Using ** patterns do not (any longer) work because deja-dub escapes [?* characters in duplicity command. See https://git.launchpad.net/deja-dup/tree/libdeja/tools/duplicity/DuplicityJob.vala#n303 :

  string escape_duplicity_path(string path)
  {
    // Duplicity paths are actually shell globs.  So we want to escape anything
    // that might fool duplicity into thinking this isn't the real path.
    // Specifically, anything in '[?*'.  Duplicity does not have escape
    // characters, so we surround each with brackets.
    string rv;
    rv = path.replace("[", "[[]");
    rv = rv.replace("?", "[?]");
    rv = rv.replace("*", "[*]");
    return rv;
  }

  void process_include_excludes()
  {
    expand_links_in_list(ref includes, true);
    expand_links_in_list(ref excludes, false);

    // We need to make sure that the most specific includes/excludes will
    // be first in the list (duplicity uses only first matched dir).  Includes
    // will be preferred if the same dir is present in both lists.
    includes.sort((CompareFunc)cmp_prefix);
    excludes.sort((CompareFunc)cmp_prefix);

    foreach (File i in includes) {
      var excludes2 = excludes.copy();
      foreach (File e in excludes2) {
        if (e.has_prefix(i)) {
          saved_argv.append("--exclude=" + escape_duplicity_path(e.get_path()));
          excludes.remove(e);
        }
      }
      saved_argv.append("--include=" + escape_duplicity_path(i.get_path()));
      //if (!i.has_prefix(slash_home_me))
      //  needs_root = true;
    }
    foreach (File e in excludes) {
      saved_argv.append("--exclude=" + escape_duplicity_path(e.get_path()));
    }

    // TODO: Figure out a more reasonable way to order regexps and files.
    // For now, just stick regexps in the end, as they are more general.
    foreach (string r in exclude_regexps) {
      saved_argv.append("--exclude=" + r);
    }

    saved_argv.append("--exclude=**");
  }

1. install dconf-editor

sudo apt install dconf-editor

2. run dconf-editor as normal user. (don't use sudo)

dconf-editor

3. locate org -> gnome -> deja-dup -> exclude-list
4. set Custom value to (replace leo with your user name)

['$TRASH', '$DOWNLOAD', '/home/leo/.anaconda', '/home/leo/**/node_modules', '/home/leo/**/__pycache__', '/home/leo/**/*.pyc']

5. You might need to reboot/re-signin. I run Screenshot which automatically update the value. I don't know why, maybe someone else can explain.

Screenshots:

Replace leo with your user name

I came up with a working workaround for this. The problem seems to be that duplicity does not itself expand wildcards (except for ** apparently) but relies on the shell to do that, and when it is run from deja-dup there is no shell involvement, which is why that now blocks configuring wildcard excludes. Sure, you can use dconf-editor to force them into the saved excludes list, but they don't work (from the monitoring in the script below I actually found that deja-dup would drop excludes containing a '*' and not pass them to duplicity at all).

To make it work we need shell expansion of the wildcards. You can do that manually and insert the results via dconf-editor, as suggested here, but this solution does it automatically at backup run time.

First find where duplicity is on your path ("which duplicity") then find a path location ahead of it in the path ("echo $PATH"). In my case it is /usr/bin/duplicity and /usr/local/bin comes ahead of that, which is perfect. Create a text file named duplicity in that latter path location (e.g. /usr/local/bin/duplicity), make it executable (chmod +x ...) and put this content in there:

#! /bin/bash

# Shim script run from deja-dup in place of duplicity, to add in file/pattern
# exclude arguments for duplicity.
#
# The excludes are read from ~/.config/deja-dup-excludes (one-per-line).

ARGS="$*"

EXCLUDES=$(cat $HOME/.config/deja-dup-excludes | sed -e 's/#.*$//' -e 's/^[ \t]*//' -e '/^$/d')

if ( echo "$ARGS" | grep -q '\--exclude'); then
    for EXCL in $EXCLUDES
    do
        EXCL_ARG=$(find $EXCL -printf '--exclude %p ')

        ARGS="$EXCL_ARG$ARGS"
    done
fi  

#echo "$ARGS" >>/tmp/dup.out

/usr/bin/duplicity $ARGS

Make sure that the last line has the correct path for the real duplicity on your machine, and you can un-comment the echo statement if you want to check your work.

Then create a file .config/deja-dup-excludes under your home directory, with the excludes listed one-per-line, e.g.:

# Exclude files/patterns for deja-dup
# (used by the /usr/local/bin/duplicity script).

/home/Ian/core.*
/etc/postfix/sasl_passwd*

Any lines beginning with a '#' will be taken as comment lines and ignored.

deja-dup will now execute that script instead of the real duplicity, and it will add in the necessary --exclude arguments before calling the latter.

A hack, admittedly, but it works a charm.