Slapd daemon can't start: TLS init def ctx failed: -1

Yesterday I had to restart my Mac OS X Lion Server and after reboot I observed that I do not have any connection to my LDAPv3 on the localhost as well as no users available in Server application.

Next, Server application has showed three outdated certificates. Their names starting with "APNS".

I am not sure whether the issue is connected to those certificates specifically, but unfortunately "slapd" refers to issues with main certificates for my internal domain.

Following is the log from slapd itself:

Oct 27 17:49:57 apple slapd[723]: @(#) $OpenLDAP: slapd 2.4.23 (Jun 24 2012 23:35:56) $
    [email protected]:/private/var/tmp/OpenLDAP/OpenLDAP-186.5~1/servers/slapd

Oct 27 17:49:57 apple slapd[723]: daemon: SLAP_SOCK_INIT: dtblsize=8192
Oct 27 17:49:57 apple slapd[723]: main: TLS init def ctx failed: -1 
Oct 27 17:49:57 apple slapd[723]: slapd stopped.

System log obviously shows:

Oct 27 17:53:19 apple com.apple.launchd[1] (org.openldap.slapd[879]): Exited with code: 1  
Oct 27 17:53:19 apple com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 10 seconds

While trying to start slapd from command line (using: /usr/libexec/slapd -d -1), I receive the following message before crash:

TLS: attempting to read
 `/etc/certificates/domain.com.456DACFFC771F8EB2F5A8E0EBB269969B8164097.key.pem'.
execv failed 
TLS: could not use key file
 `/etc/certificates/domain.com.456DACFFC771F8EB2F5A8E0EBB269969B8164097.key.pem'.
LS: error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
/SourceCache/OpenSSL098/OpenSSL098-49/src/crypto/asn1/asn1_lib.c:150
main: TLS init def ctx failed: -1 slapd destroy: freeing system resources. slapd stopped.

If I remove TLS configurations options from "/etc/openldap/slapd_macosxserver.conf" then I am able to start the daemon (using: /usr/libexec/slapd -d -1 -f slapd_macosxserver.conf. All my users a back, I can log-in via Workgroup Manager or Server application.

Meanwhile the Server Admin in the Open Directory entry shows that:

1. LDAP Server is: Running
2. Password Server is: Stopped
3. Kerberos is: Running

I have tried couple of things, but still have no idea how to fix the problem.

Solution 1:

I'd recommend creating another self-signed cert in Server.app that you can use for securing services (if the others were expired/deleted). By creating the certificate using Server.app, it will automatically be available for other services (like Open Directory).

After you've created a new self-signed certificate, follow steps 6 through 12 in this article (which describe how your SSL certificate can be configured for use with Open Directory). Performing the Open Directory -> Settings -> LDAP -> SSL configuration through Server Admin will write the correct certificate paths into the slapd config file.

Once you've corrected the certificate problems, Open Directory (slapd) should start normally (without you having to start it by hand). If Password Server still doesn't show running after that, you might try a reboot (or check to see if it's generating crash logs or other errors in Console.)

Edit

After modifying the certificate configuration for use with LDAP, it's probably worth checking to see that the machine has provided updated certificate paths to slapd in the slapd_macosxserver.conf file. That is, the unique string of numbers and characters in the key/cert paths should have changed.

To confirm that slapd can access the corresponding private key for the certificate that you're securing LDAP services with, you can check the file at /etc/openldap/slapd_macosxserver.conf...Look for a line mentioning certadmin...That line specifies the command that slapd is using to retrieve the private key from the Keychain. It's possible to perform that command (copy and paste) in Terminal to see if the private key passphrase can be retrieved:

/usr/sbin/certadmin --get-private-key-passphrase /etc/certificates/domain.com.456DACFFC771F8EB2F5A8E0EBB269969B8164097.key.pem

Once you've retrieved the passphrase, see if you can view the private key using that passphrase:

sudo openssl rsa -in /etc/certificates/domain.com.456DACFFC771F8EB2F5A8E0EBB269969B8164097.key.pem -text -noout

When prompted for the pass phrase, copy and paste the value that you obtained in the step above. You should see the private key data output on the screen. This would confirm that:

1.) Your private key passphrase can be retrieved from the Keychain

2.) The pass phrase in the Keychain can be used to decrypt the key

If you are unable to get the pass phrase and unlock the key, it's possible that slapd is not able to either. I believe that the software is using a keychain item in the System keychain named "Mac OS X Server certificate management" with a kind of "application password". The "Account" for that keychain item should be set to the same unique string of characters and numbers (456DACFFC771F8EB2F5A8E0EBB269969B8164097 in this example) that you see in the cert/key paths in /etc/certificates. If you do not see one of these corresponding application passwords in the System keychain, it may be your issue.

Solution 2:

The following has been tested on snow-leopard 10.6 and lion 10.7 versions of the operating system when OpenDirectory LDAP server will not start and Password Server is still running.

If you have a look at the tail of the file sudo tail /etc/openldap/slapd.d/cn=config.ldif you will see a set of TLS Certificate entries similar to the block below:

olcTLSCertificateFile: /etc/certificates/[myservername].ECA88C7518AEDE947AAC94D9A259D1B2E5621169.cert.pem

olcTLSCACertificateFile: /etc/certificates/[myservername].ECA88C7518AEDE947AAC94D9A259D1B2E5621169.chain.pem

olcTLSCertificateKeyFile: /etc/certificates/[myservername].ECA88C7518AEDE947AAC94D9A259D1B2E5621169.key.pem

olcTLSCertificatePassphraseTool: /usr/sbin/certadmin --get-private-key-passphrase /etc/certificates/[myservername].ECA88C7518AEDE947AAC94D9A259D1B2E5621169.key.pem

entryCSN: 20120628190714.643255Z#000000#001#000000

If you compare these entries to the values shown in /etc/certificates you will most likely find that the magic number is different for your certificate to the value shown in the /etc/certificates directory for your certificates.

If you run the command

sudo /usr/libexec/slapd -d 1

you will see towards the end of the trace a line approximating

TLS: could not load verify locations (file:`/etc/certificates…

To overcome this problem remove the last five lines from the /etc/openldap/slapd.d/cn=config.ldif file manually using your favourite editor and sudo.

Then to restart ldap issue the following commands

sudo /usr/bin/db_recover -h /var/db/openldap/openldap-data
sudo launchctl load /System/Library/LaunchDaemons/org.openldap.slapd.plist

All should then be fixed.