Automatically shrink existing Time Machine Backup to a fixed size
Time Machine will respect a size determination - but only when you make the container. This functionality is all that Apple provides for native backup so you'd need a custom setup to do what you ask, and you can't limit things once you've started backing up to a container.
Going forward if you want to make a new restricted container, the concept isn't hard to map out though.
- make a new container on the synology and be sure to enforce the size constraints when you start the process.
Server.app does that easily - I'm not aware of synology code that reverse engineered that feature, but would be surprised if it's not on GitHub or published somewhere public.
At that point, you could delete all backups of user folders that are large and then block by block copy the old backup data to the new destination and then inherit the backup.
You can also thin backup intervals from the command line or script this automation as well.
Lots and lots of work, but it might be worth it as a project if someone is inclined.
When I'm in your shoes, I just copy off the old "too large" backup to an offline hard drive and put it on a shelf and start over with new backups of the size you like. In 6 months or a year - wipe that old drive unless you really feel you need that older history.
This is the easiest solution using OSX Terminal.
Get a list of all the backups in TimeMachine. This will also show you the full directory path to the backups that you will need in step 2...
$ tmutil listbackups
/Volumes/Time Machine Backups/Backups.backupdb/{your-macbook}/2018-10-02-213405
/Volumes/Time Machine Backups/Backups.backupdb/{your-macbook}/2018-10-09-192323
/Volumes/Time Machine Backups/Backups.backupdb/{your-macbook}/2018-10-19-212659
Choose which backups to delete based on their date. Note the use of a wildcard * and the use of the directory from step 1. For example, to delete all of 2018's backups you would use this:
$sudo tmutil delete '/Volumes/Time Machine Backups/Backups.backupdb/{your-macbook}/2018-'*
The final step is to shrink and recover space from the sparse bundle. Search your backup drive for the .sparsebundle file.
$ sudo hdiutil compact '/Volumes/{your-mac}.sparsebundle'