How to identify a missing datastore for com.apple.securityd?

Resolution

a. I have a crlcache.db reference that appears dead, but no number of deletion attempts makes it go away.

This was the root problem, but it was difficult to make it go away with any of the existing tools. The crlcache.db entry was ghosted in my Keychain Access application, so an entry still existed. While I had reset all of my passwords, I hadn't fully killed the Keychain. All the applications I listed were using the Keychain to find their information, hitting crlcache.db and then either retrying or throwing I had to manually remove both of these files (essentially a hard-reset of the entire Keychain):

~/Library/Preferences/com.apple.security.plist
/Library/Preferences/com.apple.security.plist

Diagnosis

It was very difficult to diagnose the issue because nothing would tell me which file did not exist. This comment, with its command that gathered bug information for Apple, was the most helpful. This produced a giant tar.gz with lots of diagnostic goodies that told me much more about what was going on. Make sure to run along with whatever application(s) is misbehaving at the time.

sudo sysdiagnose securityd

Among the many plain text debug output files it produced, there was a large one called fs_usage.txt, and when I opened it up, I could see 1000s of familiar entries

08:01:11.999993  getattrlist                            /private/var                                                                                                                                                          0.000003   Twitter.3616
08:01:11.999996  getattrlist                            /private/var/db                                                                                                                                                       0.000003   Twitter.3616
08:01:11.999998  getattrlist                            /private/var/db/crls                                                                                                                                                  0.000003   Twitter.3616
08:01:12.000000  getattrlist            [  2]           /private/var/db/crls/crlcache.db                                                                                                                                      0.000002   Twitter.3616
08:01:12.000004  statfs64                               /private/var/db/crls      

Once I saw that, it was clear the Keychain was still the issue, and my ghosted entry had to go. Lacking the knowledge of how to perform laparoscopic surgery on the plist files, I simply amputated and started over.


In my case after removing Espionage it left a reference for its .keychains file in Keychain Access. Log entries were like these:

default 15:11:56.715938 +0200   com.apple.WebKit.Networking DbOpen of /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain
default 15:11:56.716702 +0200   com.apple.WebKit.Networking open /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain: No such file or directory
default 15:11:56.716765 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716848 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716910 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716952 +0200   com.apple.WebKit.Networking dbBlobVersion() failed for a non-existent database

The resolution was rather simple: had to remove the database reference in Keychain Access using the menu entry in File called Delete Keychain Espionage. You should try to find the missing file reference in a DbOpen line before the CSSM Exception lines to see what's missing.