How to identify a missing datastore for com.apple.securityd?
Resolution
a. I have a
crlcache.dbreference that appears dead, but no number of deletion attempts makes it go away.
This was the root problem, but it was difficult to make it go away with any of the existing tools. The crlcache.db entry was ghosted in my Keychain Access application, so an entry still existed. While I had reset all of my passwords, I hadn't fully killed the Keychain. All the applications I listed were using the Keychain to find their information, hitting crlcache.db and then either retrying or throwing I had to manually remove both of these files (essentially a hard-reset of the entire Keychain):
~/Library/Preferences/com.apple.security.plist
/Library/Preferences/com.apple.security.plist
Diagnosis
It was very difficult to diagnose the issue because nothing would tell me which file did not exist. This comment, with its command that gathered bug information for Apple, was the most helpful. This produced a giant tar.gz with lots of diagnostic goodies that told me much more about what was going on. Make sure to run along with whatever application(s) is misbehaving at the time.
sudo sysdiagnose securityd
Among the many plain text debug output files it produced, there was a large one called fs_usage.txt, and when I opened it up, I could see 1000s of familiar entries
08:01:11.999993 getattrlist /private/var 0.000003 Twitter.3616
08:01:11.999996 getattrlist /private/var/db 0.000003 Twitter.3616
08:01:11.999998 getattrlist /private/var/db/crls 0.000003 Twitter.3616
08:01:12.000000 getattrlist [ 2] /private/var/db/crls/crlcache.db 0.000002 Twitter.3616
08:01:12.000004 statfs64 /private/var/db/crls
Once I saw that, it was clear the Keychain was still the issue, and my ghosted entry had to go. Lacking the knowledge of how to perform laparoscopic surgery on the plist files, I simply amputated and started over.
In my case after removing Espionage it left a reference for its .keychains file in Keychain Access. Log entries were like these:
default 15:11:56.715938 +0200 com.apple.WebKit.Networking DbOpen of /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain
default 15:11:56.716702 +0200 com.apple.WebKit.Networking open /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain: No such file or directory
default 15:11:56.716765 +0200 com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716848 +0200 com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716910 +0200 com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716952 +0200 com.apple.WebKit.Networking dbBlobVersion() failed for a non-existent database
The resolution was rather simple: had to remove the database reference in Keychain Access using the menu entry in File called Delete Keychain Espionage. You should try to find the missing file reference in a DbOpen line before the CSSM Exception lines to see what's missing.