Combination of SSH key auth, and two-factor authentication

Recent versions of openssh include the AuthenticationMethods option:

Debian backported openssh-6.2 a while back, so I expect this to be available in Raspbian as well.

Specifies the authentication methods that must be successfully completed for a user to be granted access.

You can have the main block of your sshd_config with ChallengeResponseAuthentication enabled:

ChallengeResponseAuthentication yes
PasswordAuthentication no
PermitRootLogin no

and then use AuthenticationMethods in your Match blocks (use Group matching instead of User matching to ease scalabity):

Match Group personal
  AuthenticationMethods publickey

Match Group peon
  PasswordAuthentication yes
  AuthenticationMethods publickey,keyboard-interactive

Aditionally, you can use pam_succeed_if(8) to trigger the two-factor-authentication only if a matching group requires it:

 auth required pam_succeed_if.so quiet user ingroup peon