Is it possible to have sshd accept passwords only on certain ports?

This can be accomplished with the Match directive.

In /etc/sshd_config, declare multiple ports:

Port 22
Port 2222

Then, at the very end of the file, add the following:

Match LocalPort 2222
PasswordAuthentication no

All settings will be shared between the ports, except password authentication will be disabled on port 2222.