PBIS Open AD authentication stops working on ubuntu with errors: "user accout has expired" and "is your account locked?"
We have been using PowerBroker Identity Services Open to authenticate active directory users from ubuntu hosts successfully for six months.
Recently AD authentication stopped working on several workstations after users performed an apt-get upgrade of 200+ packages at once. Authentication attempts give the errors, "invalid password," "user account has expired," or "is your account locked?"
I have not been able to link the issue to a specific package upgrade but workstations build from scratch with the same package versions do not experience the issue. I have tried reinstalling PBIS and validated all the config files but I'm missing something.... I'm at a loss and would love any advice anyone has. I'd rather not have to rebuild another box the next time this happens!
Authentication Attempts
I first verified that the the AD user account was enabled, was not locked and had not expired. Local user authentication works fine through lightdm and ssh.
-
lightdm
- valid credentials
- error returned to user "Invalid password, please try again."
- auth.log: nothing
- syslog: nothing
-
incorrect password
- error returned to user "Invalid password, please try again."
-
auth.log:
lightdm: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:username][error code:40022] -
syslog:
lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed) lsass: [lsass] Failed to authenticate user (name = 'username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 17768
- valid credentials
-
ssh
-
valid credentials
- ssh disconnect with "Connection closed by IP_ADDRESS."
-
auth.log:
sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME sshd[18237]: error: Received disconnect from IP_ADDRESS: 13: Unable to authenticate [preauth] - syslog: nothing
-
incorrect password
- ssh disconnect with "Connection closed by IP_ADDRESS."
-
auth.log:
sshd[18276]: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:domain\username][error code:40022] sshd[18272]: error: PAM: Authentication failure for domain\\username from hostname -
syslog
lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed) lsass: [lsass] Failed to authenticate user (name = 'domain\username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 18276
-
-
Just trying some crazy local stuff (and no, the account is not locked in AD)
root@hostname:~# su - domain\\username su: Authentication failure (Ignored) reenter password for pam_mount: DOMAIN\username@hostname:~$ sudo cat /etc/fstab [sudo] password for DOMAIN\username: sudo: account validation failure, is your account locked? DOMAIN\username@hostname:~$
Configuration
- Ubuntu 14.04
- PBIS Open 8.0.1.2029 (pbis-open-8.0.1.2029.linux.x86_64.deb.sh)
-
/opt/pbis/bin/config --dump
AllowDeleteTo "" AllowReadTo "" AllowWriteTo "" MaxDiskUsage 104857600 MaxEventLifespan 90 MaxNumEvents 100000 DomainSeparator "\\" SpaceReplacement "^" EnableEventlog false Providers "ActiveDirectory" DisplayMotd false PAMLogLevel "error" UserNotAllowedError "Access denied" AssumeDefaultDomain true CreateHomeDir true CreateK5Login true SyncSystemTime true TrimUserMembership true LdapSignAndSeal false LogADNetworkConnectionEvents true NssEnumerationEnabled true NssGroupMembersQueryCacheOnly true NssUserMembershipQueryCacheOnly false RefreshUserCredentials true CacheEntryExpiry 14400 DomainManagerCheckDomainOnlineInterval 300 DomainManagerUnknownDomainCacheTimeout 3600 MachinePasswordLifespan 2592000 MemoryCacheSizeCap 0 HomeDirPrefix "/home" HomeDirTemplate "%H/%D/%U" RemoteHomeDirTemplate "" HomeDirUmask "022" LoginShellTemplate "/bin/bash" SkeletonDirs "/etc/skel" UserDomainPrefix "DOMAIN.COM" DomainManagerIgnoreAllTrusts false DomainManagerIncludeTrustsList DomainManagerExcludeTrustsList RequireMembershipOf "DOMAIN\\DOMAIN-GROUP" Local_AcceptNTLMv1 true Local_HomeDirTemplate "%H/local/%D/%U" Local_HomeDirUmask "022" Local_LoginShellTemplate "/bin/sh" Local_SkeletonDirs "/etc/skel" UserMonitorCheckInterval 1800 LsassAutostart true EventlogAutostart true -
/opt/pbis/bin/get-status
LSA Server Status: Compiled daemon version: 8.0.1.2029 Packaged product version: 8.0.2029.67662 Uptime: 1 days 1 hours 4 minutes 26 seconds [Authentication provider: lsa-activedirectory-provider] Status: Online Mode: Un-provisioned Domain: DOMAIN.COM Domain SID: S-1-5-21-3537566271-1428921453-776812789 Forest: domain.com Site: NYC Online check interval: 300 seconds [Trusted Domains: 1] [Domain: DOMAIN] DNS Domain: domain.com Netbios name: DOMAIN Forest name: domain.com Trustee DNS name: Client site name: NYC Domain SID: S-1-5-21-3537566271-1428921453-776812789 Domain GUID: 0b6b6d88-ea48-314a-8bad-a997a57bc1f4 Trust Flags: [0x001d] [0x0001 - In forest] [0x0004 - Tree root] [0x0008 - Primary] [0x0010 - Native] Trust type: Up Level Trust Attributes: [0x0000] Trust Direction: Primary Domain Trust Mode: In my forest Trust (MFT) Domain flags: [0x0001] [0x0001 - Primary] [Domain Controller (DC) Information] DC Name: dc2.nyc.domain.com DC Address: 10.x.x.50 DC Site: NYC DC Flags: [0x0000f1fc] DC Is PDC: no DC is time server: yes DC has writeable DS: yes DC is Global Catalog: yes DC is running KDC: yes [Global Catalog (GC) Information] GC Name: dc1.nyc.domain.com GC Address: 10.x.x.50 GC Site: NYC GC Flags: [0x0000f3fd] GC Is PDC: yes GC is time server: yes GC has writeable DS: yes GC is running KDC: yes -
/opt/pbis/bin/find-objects --user USERNAME
User object [1 of 1] (S-1-5-21-3537566271-1428921453-776812789-1107) ============ Enabled: yes Distinguished name: CN=USERNAME,OU=User,OU=User Accounts,DC=domain,DC=com SAM account name: username NetBIOS domain name: DOMAIN UPN: [email protected] Display Name: First Last Alias: <null> UNIX name: DOMAIN\username GECOS: First LAst Shell: /bin/bash Home directory: /home/DOMAIN/username Windows home directory: \\domain.com\dfs\NYC\Users\username Local windows home directory: UID: 1023411283 Primary group SID: S-1-5-21-3537566271-1428921453-776812789-513 Primary GID: 1023410689 Password expired: no Password never expires: yes Change password on next logon: no User can change password: yes Account disabled: no Account expired: no Account locked: no -
/etc/pbis/pbis-krb5-ad.conf
[libdefaults] default_tgs_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC default_tkt_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC preferred_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC dns_lookup_kdc = true pkinit_kdc_hostname = <DNS> pkinit_anchors = DIR:/var/lib/pbis/trusted_certs pkinit_cert_match = &&<EKU>msScLogin<PRINCIPAL> pkinit_eku_checking = kpServerAuth pkinit_win2k_require_binding = false pkinit_identities = PKCS11:/opt/pbis/lib/libpkcs11.so -
/etc/pam.d/common-session
session [default=1] pam_permit.so session requisite pam_deny.so session required pam_permit.so session optional pam_umask.so session required pam_unix.so session optional pam_mount.so session [success=ok default=ignore] pam_lsass.so session optional pam_systemd.so -
/etc/pam.d/common-auth
auth [success=2 default=ignore] pam_unix.so nullok_secure auth [success=1 default=ignore] pam_lsass.so try_first_pass auth requisite pam_deny.so auth required pam_permit.so auth optional pam_cap.so auth optional pam_mount.so -
/opt/pbis/share/pbis.pam-auth-update
Name: Likewise Default: yes Priority: 250 Conflicts: winbind Auth-Type: Primary Auth: [success=end default=ignore] pam_lsass.so try_first_pass Auth-Initial: [success=end default=ignore] pam_lsass.so Account-Type: Primary Account: [success=ok new_authtok_reqd=ok default=ignore] pam_lsass.so unknown_ok [success=end new_authtok_reqd=done default=ignore] pam_lsass.so Session-Type: Additional Session: sufficient pam_lsass.so Password-Type: Primary Password: [success=end default=ignore] pam_lsass.so use_authtok try_first_pass Password-Initial: [success=end default=ignore] pam_lsass.so -
/usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf
[SeatDefaults] user-session=ubuntu greeter-show-manual-login=true -
/usr/share/lightdm/lightdm.conf.d/50-unity-greeter.conf
[SeatDefaults] allow-guest=false greeter-show-remote-login=false greeter-show-manual-login=true greeter-session=unity-greeter
Solution 1:
The key line is this:
sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME
This indicates that a PAM module believes the account has expired. I'd focus less on auth/session and more on account, which is the facility focused on account properties not related to authentication. Your first task is to identify the module causing the problem. Once you know that, it should be much easier to identify why the module thinks the user should be blocked.
Review the applicable account modules one by one, and try adding the debug flag to individual entries to expand the logging output if you need more hints. If truly stumped and it wouldn't violate the security of a critical environment, you can also try commenting the account lines one at a time until you identify your culprit.
As for what changed, more than likely your PAM config was modified when these packages were installed. Chances are that the users in question were in this state all along, but the database associated with the misbehaving account module was being bypassed. (skipped, commented, not present at all, etc.)
Solution 2:
FYI: domainjoin-cli configure --enable pam will re-add these lines after an upgrade as well. PBIS Open 8.x and higher properly deliver a /usr/share/pam-configs/pbis configuration so that this shouldn't happen in the future.
Additionally, PBIS logs more specific errors to the daemon facility of syslog, so you can view them in ubuntu in /var/log/syslog rather than /var/log/secure.
Solution 3:
Thanks again to @Andrew B for helping me find the solution.
For the record, here is a copy of the correct /etc/pam.d/common-account configuration that FIXES my issue (the two pam_lsass.so lines were missing from the systems that were not working):
account [success=3 new_authtok_reqd=done default=ignore] pam_unix.so
account [success=ok new_authtok_reqd=ok default=ignore] pam_lsass.so unknown_ok
account [success=1 new_authtok_reqd=done default=ignore] pam_lsass.so
account requisite pam_deny.so
account required pam_permit.so