Allow SSH only on one of a servers IP addresses

Take a look at ListenAddress in your sshd_config.

ListenAddress
         Specifies the local addresses sshd(8) should listen on.  The
         following forms may be used:

               ListenAddress host|IPv4_addr|IPv6_addr
               ListenAddress host|IPv4_addr:port
               ListenAddress [host|IPv6_addr]:port

         If port is not specified, sshd will listen on the address and all
         prior Port options specified.  The default is to listen on all
         local addresses.  Multiple ListenAddress options are permitted.
         Additionally, any Port options must precede this option for non-
         port qualified addresses.