Fixing starttls verify=fail, verifymsg=unable to get local issuer certificate

Running Amazon Linux on EC2 instance with sendmail. I have an email account with Network Solutions, and use that account as a SMART_HOST relay in my sendmail configuration. It works well except for one little detail.

In my maillog file I see entries like this:

sendmail[28450]: STARTTLS=client, relay=mail.example.com.netsolmail.net., version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256

After a little research, I've come to the conclusion that the verify=FAIL is essentially harmless: the connection actually was encrypted, it's just that the host's certificate could not be verified.

Since nobody but me reads the log file, I wouldn't care. But when the message arrives, the Received header shows

Received: from unknown (HELO example.com) ([email protected]@12.34.56.78)
  by 0 with ESMTPA; 15 Aug 2016 07:10:15 -0000

I was hoping to see with ESMPTSA but I would guess that the certificate verification failure caused the 'S' to be surpressed.

How can I get more detail on what was wrong with the certificate, and how can avoid the verification failure? My guess is that the multiple subdomains of mail.example.com.netsolmail.net don't match closely enough with the name on the certificate. But how can I verify that, and how can I avoid the complaint - or more exactly how can I get the Received header to acknowledge the secure connection with ESMTPSA.

EDIT: I edited sendmail.mc to add

define(`confLOG_LEVEL', `15')dnl

Now maillog gives more details. Right after the verify=FAIL line I now see:

sendmail[30706]: STARTTLS=client, cert-subject=/OU=GT39680792/OU=See+20www.rapidssl.com/resources/cps+20+28c+2915/OU=Domain+20Control+20Validated+20-+20RapidSSL+28R+29/CN=*.hostingplatform.com, cert-issuer=/C=US/O=GeoTrust+20Inc./CN=RapidSSL+20SHA256+20CA+20-+20G3, verifymsg=unable to get local issuer certificate

I take this to mean that at least one cause of the verification failure is that sendmail can't find a certificate for the local machine it's running on? Since I'm only relaying outgoing mail to a netsol server, never accepting incoming mail from the internet, I didn't think I'd need to have a certificate for this server. If I need one, where/how do I install it? And can it be the same certificate I use for my webserver, or do I need a different one? Would use of a self-signed certificate be good enough to get the Received header to say with ESMTPSA, or would it need to be a commercial certificate from a CA?

EDIT #2:

I'm accepting the answer by @MadHatter. The key was getting confCACERT defined. I'm embarrassed, my only excuse is old senile brain not grocking m4 source. The default sendmail.mc file on Amazon Linux already had

define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl

in it, and I had verified that the file existed. But I failed to notice the sneaky little dnl that was actually at the beginning of those lines! I know what it means, but since I very rarely look at m4 source, and it was right after some other dnl-ed lines that were marked as comments with #, my brain registered them as not being commented out!

I actually went through a bunch of gyrations downloading certs from Firefox and pointing sendmail at the Digicert certificate that I use for our website, but since this host only ever sends, never receives, email, nothing else was necessary. I put back the dnl on the defines for confSERVER_CERT and confSERVER_KEY, and all was well, with maillog showing verify=OK and verifymsg=ok on the appropriate STARTTLS=client lines.

But even though there were no diagnostics about TLS, the Received header for the connection to netsol still shows with ESMTPA and not with ESMTPSA. Oh well, @MadHatter had the dope on that one, too. Sorry this was so long and sort of a wild goose chase. But I learned a lot, and I did improve my configuration (in a non-vital way). I hope someone desperate enough to wade through this might learn something, too.

This is very much a question in parts, and I'll take it as such. I've used sendmail as my preferred MTA for some decades, but I can't claim to be an expert in it (ie, I'm not Eric Allman).

verifymsg=unable to get local issuer certificate

I take this to mean that at least one cause of the verification failure is that sendmail can't find a certificate for the local machine it's running on?

This seems to be an OpenSSL message, rather than an MTA one, and as I understand it it means that the verifying app is unable to get local copy of the issuer's certificate. In other words, sendmail doesn't have access to a certificate bundle that includes the root certificate for whoever issued the remote server's certificate. Remember that Linux doesn't provide a centralised certificate management service, so each app must roll its own bundle. In my case, I gave sendmail access to a certificate bundle with the following m4 code:

define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.with.intermediate.crt')dnl

I was hoping to see with ESMPTSA but I would guess that the certificate verification failure caused the 'S' to be surpressed.

I think that guess is wrong. RFC2821 and 2822 are pretty elastic about the format of a Received: header, and I cannot find anything therein that distinguishes between ESMTPA and ESMPTSA (sic). My headers all show lines such as:

Received: from example.com (example.com [80.70.90.61])
    by lory.teaparty.net (8.14.4/8.14.4) with ESMTP id u6G25OXi006577
    (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL)
    for <[email protected]>; Sat, 16 Jul 2016 03:05:24 +0100

To understand what your receiver's MTA means by ESMTPA, you'd have to find out what that MTA is, and read the documentation. Until you've done that, I don't think you can read much into that collection of letters.

Can I avoid starttls verify=fail when server hostname does not match certificate

I don't think you can. The fundmental idea behind SSL is that (a) the hostname you connect to (b) offers a certificate signed by a trusted third-party (c) with that hostname in either the CN or an SAN field. If any of those properties is not met, it is right that SSL notes that it can't verify the identity of the peer.

You shouldn't read too much into it; self-signed certificates are still very common in email handling. Of my last 1919 TLS-secured emails sent/received, 1764 involved a peer whose identity could not be verified for some reason, whilst only 155 could. You yourself are running with a self-signed certificate; you should therefore be happy that most people don't really care about the trust-chain on SMTP TLS peers!