ddev pull command seems to wipe/clear/empty the .downloads folder which results in rsync being redundant

Not sure if I'm doing something wrong but here is my situation. I'm trying to setup a 'provider' to download a database dump and files folder from a remote server, this is working in the most part. The database download works and imports perfectly well.

My issue is with the files sync. My files folder is around 5GB and with user generated content daily this only goes up steadily. I was hoping that by using rsync I could get an initial download of the files folder into the .download folder and then when running ddev pull XXXX the rsync would simply top up the folder with any missing files and save downloading the entire folder every time.

The problem I am finding is that ddev pull seems to clear the .downloads folder once initiated. This doesn't seem to be handled within the provider.yaml file but within the ddev pull.go or associated functions.

Below is my provider.yaml contents.

environment_variables:
  dburl: [email protected]:/mnt/backups/db.sql.gz
  filesurl: [email protected]:/var/www/sites/default/files

auth_command:
  command: |
    set -eu -o pipefail
    ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )

db_pull_command:
  command: |
    set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    rsync -az "${dburl}" /var/www/html/.ddev/.downloads/db.sql.gz
  service: web

files_pull_command:
  command: |
    set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null
    rsync -avz --size-only --ignore-existing "${filesurl}" /var/www/html/.ddev/.downloads
  service: web

Any help would be great! Many thanks, Mark

You can use your own strategy with files these days (I think since ddev v1.18.2). You don't have to put them in the place ddev expects or anything. Try just using the files_import_command and do whatever you want in that, including the import. Rsync directly to your target files directory. You're in charge. Omit the files_pull_command.

For more detail, see https://ddev.readthedocs.io/en/latest/users/providers/provider-introduction/

The git provider may be a decent example for you, https://github.com/drud/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/git.yaml.example

A PR to update the rsync example would be appreciated when you get this sorted out, because I think you're on track for a better technique.