Error "Can't assign requested address" when ssh-ing to remote server

On ubuntu 14.04 I have installed openssh, changed the ssh port. I can login within LAN with no problems, but I cannot access it from outside.

I also have: allowed it though firewall and set up port forwarding on my linksys router

Whenever I login, I get the error (I have replaced # instead of numbers):

ssh: connect to host #.#.#.# port #: Can't assign requested address

What else can I check, what could be wrong? Please help.

This might not fix your issue but I had the same error on OSX after it working successfully before. Flushing the routing table helped:

sudo ifconfig en0 down # take the networking interface down
sudo route flush # flush the route table
sudo ifconfig en0 up # take the interface back online

I ran these on the computer that I was connecting from. Reference: http://codefromabove.com/quickies/osx-cant-assign-requested-address-code49/