Bash: Dynamically traffic shaping rsync using variables for times schedule and slowing during runtime?

What are some solutions to having dynamic traffic shaping run during a bash script, after a command has already started? Is this even possible?

My use case is, I have rsync running on a huge set of disks to a remote backup location and this takes many many hours over Internet, so I'd like to apply traffic shaping to the line where rsync is invoked, but only during specified times.

So say for instance, current schedule is to reduce uploads to 1000 kilobytes per second (1.0MB/s) during 5am to 10am AND 3pm to 9pm.

I've looked at --bwlimit=1000 for rsync but this shapes rsync for the entire time it runs, not just the desired shaping timeframes.

Is it possible to slow rsync once it's started? Because if that's possible somehow, that would be my solution.

I've seen people suggest wondershaper but I'm wondering if this could be "switched on and off" programmatically? And if yes, how to do that?

Here's a mock up that I've hacked together using generous contributions from Stackoverflow, that gives the functionality of specifying and checking a time schedule (based on the hour, which is fine for my needs at the moment)...

#!/bin/bash

declare -A shapingTimes
declare -A keycount

useTrafficShaping=1

# schedule
shapingTimes=([0-from]=5 [0-to]=10)  # 5am to 10am
shapingTimes+=([1-from]=15 [1-to]=21)  # 3pm to 9pm

# because keys and array content differs in order from the way they are defined, we need to tidy this up by setting up the shaping times order in the way in which it is defined above
# to do this, count how many keys used in array entry (https://stackoverflow.com/questions/63532910)
# we use this result to to dynamically calculate the total number of entries ($totalshapingTimes)
for i in "${!shapingTimes[@]}"; do keycount[${i//[0-9]-/}]=""; done
totalshapingTimes=$((${#shapingTimes[*]} / ${#keycount[@]}))
x=1; while [[ $x -le "$totalshapingTimes" ]]; do shapingTimes_order+="$(( x-1 )) "; x=$(( $x + 1 )); done
# 'shapingTimes_order' array is used later in this script to process which shapingTimes are handled in what order


if [[ -n $useTrafficShaping ]] && [[ $useTrafficShaping == 1 ]]; then
    echo "Traffic shaping: ON"
    # get current time (hour) to determine if we're inside peak times
    currentHour=$(date +%H)
        
        # display our traffic shaping time windows as defined in above array
        echo "Defined schedule:"
        for i in ${shapingTimes_order[*]}; do
            echo "${shapingTimes[$i-from]}hrs to ${shapingTimes[$i-to]}hrs"
            # work out which peak times we're using, thanks to the help of Glenn Jackman (https://stackoverflow.com/questions/18128573)
            if (( ${shapingTimes[$i-from]} <= 10#$currentHour && 10#$currentHour < ${shapingTimes[$i-to]} )); then
                # current time matches a specified shaping timeframe, so rsync should go SLOW!
                shape=1
            fi
        done

        echo "The current hour is $currentHour"
        if [[ -z $shape ]]; then
            echo "Not in a shaping window, rsync can run as normal."
            # some command here, run rsync as normal
        else
            echo "Matches a shaping schedule. *** SHAPING NETWORK TRAFFIC ***"
            # command here to kick in traffic shaping
        fi

else
    echo "Traffic shaping: OFF"
    # run rsync as normal
fi

rsync cannot have its bandwidth limit parameter changed dynamically.
However you could

  • Split the job into several smaller tasks where each will have a specific bw limit based on the time it starts, eg

for instance

rsync a b c d e target:/dir/

to

rsync --bw-limit=bw1 a target:/dir/
...
rsync --bw-limit=bw2 e target:/dir/
  • Use the --time-limit=MINS option

rsync is smart and will not redo what it just did, if the files are synced, so you could run rsync for 1 hour with a bw limit bw1, then restart it with bw2, etc (and pause it if it needs to be), or combine with the above solution.

rsync --bw-limit=bw1 --time-limit=60 ...

rsync --bw-limit=bw2 --time-limit=60 ...
  • Change rsync

If you feel like it, rsync is open-source, and you could add some code to have rsync answer dynamically to some signal (signal and kill) that would change the internal options (maight have to change the target rsync daemon as well depending on the code)

rsync --my-new-version ...

kill -SIGNALTOSLOW rsyncpid
...
kill -SIGNALTOSPEED rsyncpid