How to restrict an SSH tunnel with SOCKS v5 to only a certain outbound IP address on an SSH server? (Debian)

I do find a lot of information of how to limit incoming access to certain IP addresses, but not do do it on the other side, the outbound side. When i have a SOCKS 5 Proxy. I'm primary looking for a way to do it in openssh (ssh_config) but if not possible then with IP-Tables.

Assuming standard OpenSSH server I've been through the docs and the closest looks to be the PermitOpen option.

 PermitOpen
         Specifies the destinations to which TCP port forwarding is per‐
         mitted.  The forwarding specification must be one of the fol‐
         lowing forms:

               PermitOpen host:port
               PermitOpen IPv4_addr:port
               PermitOpen [IPv6_addr]:port

         Multiple forwards may be specified by separating them with
         whitespace.  An argument of any can be used to remove all re‐
         strictions and permit any forwarding requests.  An argument of
         none can be used to prohibit all forwarding requests.  The
         wildcard ‘*’ can be used for host or port to allow all hosts or
         ports respectively.  Otherwise, no pattern matching or address
         lookups are performed on supplied names.  By default all port
         forwarding requests are permitted.

To restrict the forwarding for only certain users, you could use something like

Match User user1 ...
PermitOpen ...

Trying to do this with iptables isn't going to give you the control you want as all traffic will look to originate from the openssh server process and from the proxy machine with no way to identify which user had requested it.