Why is SSH password authentication a security risk?

Most guides for OpenSSH configuration advise to disable password authentication in favor of key-based authentication. But in my opinion password authentication has a significant advantage: an ability to connect from absolutely anywhere without a key. If used always with a strong password, this should not be a security risk. Or should it?

There are pro's and con's for either pw or key-based authentication.

In some cases, for example, key-based authentication is less secure than password authentication. In other cases, its pw-based that's less secure. In some cases, one is more convenient, in others, less.

It all boils down to this: When you do key-based authentication, you must secure your key with a passphrase. Unless you have ssh-agent running (ssh-agent frees you from entering your passphrase every time), you've gained nothing in terms of convenience. Security is disputable: the attack vector now shifted from the server to YOU, or your account, or your personal machine, (...) - those may or may not be easier to break.

Think outside of the box when deciding this. Whether you gain or loose in terms of security depends on the rest of your environment and other measures.

edit: Oh, just saw that you're talking about a home server. I was in the same situation, "password" or "USB stick with key on it" always with me? I went for the former but changed the SSH listening port to something different than 22. That stops all those lame script kiddies brute forcing whole network ranges.

Using ssh keys do have one unique feature compared to password login: you can specify the allowed commands. This can be done by modifying ~/.ssh/authorized_keys file at the server.

For example,

command="/usr/local/bin/your_backup_script.sh", ssh-rsa auiosfSAFfAFDFJL1234214DFAfDFa...

would allow only the command `/usr/local/bin/your_backup_script.sh" with that particular key.

You can also specify the allowed hosts for the key:

from="yourclient,yourotherclient", ssh-rsa auiosfSAFfAFDFJL1234214DFAfDFa...

Or combine the two:

from="yourbackupserver", command="/usr/local/bin/your_backup_script.sh", ssh-rsa auiosfSAFfAFDFJL1234214DFAfDFa...

With keys you can also grant temporary access to some user (say, a consultant) to a server without revealing the password for that particular account. After the consultant has finished his/her job, the temporary key can be removed.

You can get the best of both worlds by only allowing password authentication from within your network. Add the following to the end of your sshd_config:

PasswordAuthentication no
Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
    PasswordAuthentication yes