SSH public key authentication with google authenticator still asks for password

I'm trying to enable 2FA with ssh using libpam-google-authenticator. Not all users need authenticator enabled. Everybody uses ssh public keys, and nobody has a password. I'm running Debian buster, and I've also tried libpam-google-authenticator from bullseye.

My problem is that no matter what I put in the PAM config, users without authenticator enabled are never logged straight in, but always asked for a password.

I've install libpam-google-authenticator and configured /etc/ssh/sshd_config with:

PasswordAuthentication no
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
PasswordAuthentication no
PermitEmptyPasswords no

I haven't been able to work out the correct PAM config so that users without a .google_authenticator file are still logged in. Depending on what I use, users are either prompted for a password (they don't have one), or not allowed in at all.

In /etc/pam.d/sshd I've tried (like this Trying to get SSH with public key (no password) + google authenticator working on Ubuntu 14.04.1):

#@include common-auth
auth       required     pam_google_authenticator.so debug nullok

In this case, users without an authenticator setup get rejected with the following debug;

Aug 05 15:11:18 <host> sshd(pam_google_authenticator)[746624]: debug: start of google_authenticator for "<user>"
Aug 05 15:11:18 <host> sshd(pam_google_authenticator)[746624]: debug: end of google_authenticator for "<user>" Result: The return value should be ignored by PAM dispatch
Aug 05 15:11:18 <host> sshd[746620]: error: PAM: Permission denied for <user> from <IP>

Is pam_permit is needed to set up the fallback case?

I've also tried various combinations of auth required and auth sufficient before and after @include common-auth but they all result in users without authenticator being asked for a password and sometimes users WITH authenticator also being asked for a password.

Does anyone have a recipe to make this work?

Here is my working configuration. Some users have authenticator enabled and some don't, and only SSH logins with public keys are permitted, never passwords.

In /etc/ssh/sshd_config,

UsePAM yes
PasswordAuthentication no
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
PermitEmptyPasswords no

In /etc/pam.d/sshd,

# Standard Un*x authentication.
#@include common-auth

# Require authenticator, if not configured then allow
auth    required    pam_google_authenticator.so debug nullok
auth    required    pam_permit.so

@include comon-auth must be disabled because it includes pam_unix, which I don't want to use. Then you need pam_permit to make authentication successful for users without authenticator (for which pam_google_authenticator returns ignore rather than pass).

This still doesn't let root login with an ssh key; sshd logs

sshd[1244501]: fatal: Internal error: PAM auth succeeded when it should have failed

This is discussed at Google Authenticator PAM on SSH blocks root login without 2FA .

Having gotten this working as above, I think it's actually nicer to enforce 2FA for certain groups using the SSH config as @zoredache suggested. This easily allows you to whitelist certain IPs as not requiring 2FA also. In this case, sshd_config says for example

UsePAM yes
PasswordAuthentication no
ChallengeResponseAuthentication yes
#AuthenticationMethods any # default
PermitEmptyPasswords no

Match Group adm Address *,!172.16.1.0/24
    AuthenticationMethods publickey,keyboard-interactive

and /etc/pam.d/ssh says

 Standard Un*x authentication.
#@include common-auth

# Require authenticator; SSH should not allow any user in who doesn't have it
auth       sufficient   pam_google_authenticator.so debug nullok
auth       requisite    pam_deny.so