Restrict SSH login from a specific user to a specific IP address
In addition to the wrappers option....I imagine this rsync backup is making use of an ssh key. You can restrict a key to a specific source IP or domain. This would be equivalent to a user-to-IP restriction since only that user is making use of that key (if your smart, which you appear).
First line of authorized_keys file:
from="trusted.domain.com",no-port-forwarding,no-pty ssh-rsa AAAABasdf