MIT Kerberos keeps asking for password when authenticating to OpenSSH

I am trying to setup a simple Kerberos environment which consists of a Kerberos server (KDC), a client machine and a server machine running an OpenSSH daemon. The client is supposed to be authenticated through Kerberos when establishing an SSH connection to the server machine.

The name of the Kerberos user that I'm trying to authenticate is krbuser. This user exists on the service machine and has a uid of 1001. The strange thing is that I need to enter the Kerberos user's password when logging in with SSH. Each time I login. Not only for my first connection. This seems strange because the whole point of Kerberos is to authenticate without a password.

I took a tcpdump during the authentication process and noticed that the client is doing AS-REQs to the KDC with cname root. This Kerberos username does not and I have no idea why the client is using this name. As expected, the KDC responds with a eRR-C-PRINCIPAL-UNKNOWN message, since there is no root user in the database.

To me it seems that the main issue is the fact that the client tries to authenticate as root instead of krbuser.

I'll post some information about my current configuration below. Please let me know if you need any additional information.

On the KDC:

/etc/krb5.conf

[logging]
    default = FILE:/usr/local/krb5/var/log/krb5lib.log 
    kdc = FILE:/usr/local/krb5/var/log/krb5kdc.log
    admin_server = FILE:/usr/local/krb5/var/log/kadmin.log

[libdefaults]
    default_realm = metz.prac.os3.nl
    rdns = false

# The following krb5.conf variables are only for MIT Kerberos.
    krb4_config = /etc/krb.conf
    krb4_realms = /etc/krb.realms
    kdc_timesync = 1
    ccache_type = 4
    forwardable = true
    proxiable = true

[realms]
    metz.prac.os3.nl = {
        kdc = krb-0.metz.prac.os3.nl
        admin_server = krb-0.metz.prac.os3.nl
    }

On the service machine:

/etc/ssh/sshd config (an extract)

# Kerberos options
KerberosAuthentication yes
# KerberosGetAFSToken no
# KerberosOrLocalPasswd no
# KerberosTicketCleanup yes

# GSSAPI options
GSSAPIAuthentication yes
# GSSAPICleanupCredentials yes

Captured log files during the SSH authentication:

debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "218.65.30.30"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: inetd sockets after dupping: 5, 5
Connection from 145.100.110.115 port 51946 on 145.100.110.116 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: permanently_set_uid: 106/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: [email protected] [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user krbuser service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "krbuser"
debug1: PAM: setting PAM_RHOST to "145.100.110.115"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user krbuser service ssh-connection method gssapi-with-mic [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 2 failures 1 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 3 failures 2 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
debug1: userauth-request for user krbuser service ssh-connection method password [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 1001/1001 (e=0/0)
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 1001/1001 (e=0/0)
debug1: restore_uid: 0/0
debug1: do_pam_account: called
Accepted password for krbuser from 145.100.110.115 port 51946 ssh2
debug1: monitor_child_preauth: krbuser has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: temporarily_use_uid: 1001/1001 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: PAM: establishing credentials
User child is on pid 20617
debug1: SELinux support disabled
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1001/1001
debug1: rekey after 134217728 blocks
debug1: rekey after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: SELinux support disabled
debug1: session_pty_req: session 0 alloc /dev/pts/2
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on pts/2 for krbuser from 145.100.110.115 port 51946 id 0
debug1: Setting controlling tty using TIOCSCTTY.
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 4 failures 3 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 5 failures 4 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 6 failures 5 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: Kerberos password authentication failed: Client '[email protected]' not found in Kerberos database
debug1: krb5_cleanup_proc called
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 218.65.30.30 port 18460 ssh2
maximum authentication attempts exceeded for root from 218.65.30.30 port 18460 ssh2 [preauth]
Disconnecting: Too many authentication failures [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 20604
debug1: audit_event: unhandled event 12
debug1: inetd sockets after dupping: 5, 5
Connection from 218.65.30.30 port 58146 on 145.100.110.116 port 22
debug1: Client protocol version 2.0; client software version nsssh2_4.0.0032 NetSarang Computer, Inc.
debug1: no match: nsssh2_4.0.0032 NetSarang Computer, Inc.
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: permanently_set_uid: 106/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: diffie-hellman-group14-sha1 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEXDH_INIT [preauth]

On the client machine:

kinit and klist before authentication:

$ kinit -p krbuser
$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: [email protected]

Valid starting     Expires            Service principal
06/15/17 00:24:05  06/15/17 10:24:05  krbtgt/[email protected]
    renew until 06/16/17 00:23:56

/etc/ssh/ssh config (an extract):

    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no

And finally the SSH authentication:

$ ssh [email protected] -vvv
...
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "service-0.metz.prac.os3.nl" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to service-0.metz.prac.os3.nl [145.***.***.***] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/client/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
...
debug1: Found key in /home/client/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/client/.ssh/id_rsa ((nil))
debug2: key: /home/client/.ssh/id_dsa ((nil))
debug2: key: /home/client/.ssh/id_ecdsa ((nil))
debug2: key: /home/client/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
...
debug1: Next authentication method: password
[email protected]'s password: <entering PW of Kerberos user> !!!
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to service-0.metz.prac.os3.nl ([145.***.***.***]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
...
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Thu Jun 15 00:28:57 2017

Connection established

Now klist shows the following tickets:

$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: [email protected]

Valid starting     Expires            Service principal
06/15/17 00:24:05  06/15/17 10:24:05  krbtgt/[email protected]
    renew until 06/16/17 00:23:56
06/15/17 00:27:37  06/15/17 10:24:05  host/service-0.metz.prac.os3.nl@
    renew until 06/16/17 00:23:56
06/15/17 00:27:37  06/15/17 10:24:05  host/[email protected]
    renew until 06/16/17 00:23:56

So, to sum up: The Kerberos user on the client can establish an SSH session to the server by entering its Kerberos (!) password. Not its UNIX password. The tcpdump showed that the client authenticates as root which is not a Kerberos user and I have no idea why it is using this username instead of krbuser. Which is the one I used with the kinit command.

Can anyone tell me why this authentication is not working correctly? Please let me know if you need additional information. I tried to keep it short.

Solution 1:

Root login

Judging from your supplied log files I would say that the attempted root login is not your issue here. Notice that all of the authentication attempts as root come from 218.65.30.30, a simple whois command tells us this is an IP from China. The other IPs that are successfully involved in this authentication 145.100.110.11{5,6} both come from the same (university administrated) netblock. I assume that these are your machines of interest. I would also very much recommend installing something like fail2ban that blocks IPs that are trying to brute force you.

SSH

Now let's get on to solving your Kerberos problem. I recently set up a Kerberos environment, this was fraught with difficulty so I feel your pain! Your SSH configuration looks fine so we can rule that out. Though from a security perspective you may want to consider, is setting GSSAPIDelegateCredentials yes in your /etc/ssh/ssh_config is actually required? While convenient, this option will expose your TGT to any machine you SSH into.

Keberos Caveats

Two of the main things that kerberos setups are sensitive to are time skew and name resolution. I think time skew is unlikely to be your problem here. It would be easier to confirm this with your kerberos logs though. Most modern GNU/Linux distributions come with network time already set up. You can verify this using something like timedatectl. You should also make sure that forward and reverse DNS is set up correctly for each host in your environment.

An aside about realms - while not always strictly required, the convention is to keep your realm in UPPERCASE. This is an absolute must when you're working in mixed GNU/Linux and Windows environments sometimes this will result in total failure to work. If you stick to this in general you will make life easier for yourself.

Do hostnames match DNS names and keytab names?

The major thing that stands out to me most here is the service ticket you have with the empty realm i.e. from SPN host/service-0.metz.prac.os3.nl@. To me, this smells like something funky is happening with your network set up. It looks like your default_realm is set correctly so I think there is an issue at the end hosts. To debug this I would recommend starting a new instance of sshd on the service with extra logging (sudo /usr/sbin/sshd -p 9001 -D -dd), then try to connect from the client (ssh -p 9001 -vv [email protected]). This will show you from both sides exactly what is happening during authentication and which SPNs are actually being used.

I had a problem recently where authentication failed because the service I was connecting to did not know its own FQDN (i.e. service tried to find keytab entry for host/service-0@REALM instead of host/service-0.rest.of.domain@REALM). You can check this using hostname --fqdn, if this doesn't return the full DNS name of the system then you can edit /etc/hosts. Make sure that the output of hostname --fqdn matches what's in the DNS.

Finally, make sure that you have a keytab file for each host that has the correct SPN. This should look like host/service-0.rest.of.domain@REALM. You can check the contents of your keytab file using klist -k /path/to/keytab. Happy debugging! I know how frustrating Kerberos can be.

The nuclear option

If you are still very early in your set up phase, it may just be easier to kill your setup with fire and start from scratch :) . Keeping all of this in mind and following a good guide may save you some time in the long run.

Solution 2:

There are several things to consider:

1. Can you get the service ticket on your client machine?

Verify that you can get the ticket for the service you want to use. For SSH this is usually host/@REALM or in your case host/[email protected]. If you run kvno host/[email protected] on the client machine (where you also run ssh) it should tell you a key version number and the ticket should show up in your klist.

2. Did you get time-sync right?

Make sure both machines have the same time and timezone (otherwise the time may be the same, but in a different timezone). It is probably easiest to set up NTP on all machines doing kerberos

3. Did you add the host's principals to /etc/krb5.keytab?

On the machine you're connecting to you must have entries for the host-principal (host/@REALM) with the right key-version-number (the one you determined in step 1). The file must also be readable by sshd

4. Did you get your basic Networking right?

Kerberos really loves forward DNS and reverse DNS. In fact most weird issues with Kerberos I had to debug so far had to do with broken DNS resolution, reverse-DNS not pointing to the hosts FQDN, misconfigured hostnames, etc.

Make sure all hosts can resolve all other hosts forward and reverse. Also make sure the hostname is configured correctly (with domainname) on all machines. The commands hostname, hostname -f, hostname -s and hostname -i should return sensible results.

Having said all that: please consider renaming your realm. The de-facto standard for kerberos realms is ALL.CAPS. So when you rebuild your lab or start with your production environment it should be ALL.IN.UPPER.CASE, so "METZ.PRAC.OS3.NL". This is just a convention, but maybe someone relies on that. Also you can now differentiate your DNS domain from your realm by just looking at it :)