How to mitigate STARTTLS MITM (downgrading and forged certificates) between email servers?

I'm not as technically inclined as most on this site so please keep that in mind. I wanted to learn more about email security so I did some research and everything is according to my understanding, so please correct me wherever needed. The connection between client and server (MSA/MTA?) can be encrypted but encryption between server to server (MTA to MTA?) depends on each server supporting STARTTLS. STARTTLS can also be implemented between client and server.

The only weakness in STARTTLS that I came across is that the connection can be accepted over encryption or as plain text so it makes a MITM attack supposedly trivial where the encryption can be downgraded to plain text. Also I understand there's an issue with certificates being faked, although I don't know if that's directly tied to the MITM.

I read that this is an issue with implementation rather than the protocol itself. Some solutions I came across involved configuring the client to either notify when the connection doesn't happen over SSL/TLS or to drop the connection completely. Is there a proper way to implement STARTTLS so that MITM is not possible (rather than reacting to it when it happens, if that makes sense) and where would that take place? Are these fixes intended to cover the server to server connection or just client to server? I'd also like to know a bit more about the problem with fake certificates, how that weakness is possible and how to address it, either through implementation or some other means.

I'm mostly interested in the security between server to server since server to client seems to be less of an issue and already taken care of by most ESP's. Is there a better alternative to STARTTLS? As I mentioned I'm new to all this and I'll need comprehensive instruction on most of this, including how to implement/configure. If noone can provide that here I'd appreciate any point in the right direction as far as learning resources go.

ON THE SIDE... I'm also interested in learning about actual attacks/exploits and I was wondering if the kind of MITM (server to server) mentioned above is something that can be targeted to a specific connection (destination email address?) or is more akin to grabbing whatever happens to passby.

Solution 1:

There are several problems with the transporting email a secure way and this questions might be better asked at security.stackexchange.com. The mail can be intercepted at various stages:

• Explicit TLS (STARTTLS) and implicit TLS (SMTPS) provide hop-by-hop and not end-to-end security, that is each mail server in between has access to the unencrypted mail.
• The next hop for a mail is determined by looking up the MX record via DNS. This can be spoofed unless DNSSec is used, which is mostly not the case. If it is spoofed even proper checking of the certificates within the TLS handshake will not help, because it will check that the connection to the wrong MTA is the correct one.
• MX is only defined to use the server at port 25, that is with explicit TLS (STARTTLS). Since there is no way for the sending MTA to know if the receiving MTA even supports STARTTLS the connection can be reduced to plain by a man-in-the-middle attacker by modifying the list of supported extension in the reply to the EHLO command or by denying the STARTTLS command. And even hacker with no way to intercept the connection directly might be able to inject TCP RST packets at the beginning of the TLS handshake so that the client thinks that there are problems using TLS and will fall back to plain text. Since there are actually enough real-world problems with the TLS handshake the client might not get suspicious.
• And within the TLS handshake often no proper validation is done. Sometimes no certificates gets checked at all, sometimes only the chain but not if the hostname matches etc. But again, if MX is already spoofed validation might be done against the wrong server anyway :(

Which means to make at least hop-by-hop encryption safe enough you have to add non-trivial fixes to multiple places and most of them are not in control of a sending MTA.

To get in detail to questions from the comments:

1.) Is STARTTLS normally implemented between servers or does it start from client-server and proceed from hop to hop?

STARTTLS must be implemented on each hop (MTA). It's can be used only if the receiving MTA does support STARTTLS and the sending MTA can do it. It does not depend on how the mail was delivered to the initial hop (MTA) from the client. Generally TLS with SMTP only encrypts the connection between hops, so it can be read in clear on each hop if it is not encrypted end-to-end too (with PGP or S/MIME).

2.) Can mail encrypted by PGP or S/ MIME still be redirected by MX spoofing?

Yes, it can still be redirected. But the mail itself can still only be decrypted by the real recipient, i.e. the owner of the target key.

3.) Does DNSSEC have to be adopted on a wide scale to be useful or is it something an individual can benefit from by configuring on his own email server?

Every bit helps, but only a bit. And note that it not only needs the DNSSec records but the forwarding MTA must actually use DNSSec. Most MTA probably do only DNS and don't realize if they get a spoofed MX record.

4.) Do you have to worry about TCP RST injection for implicit tls or just explicit?

Implicit TLS is only used from client to MTA, i.e. to the initial hop. This is the most secure step because it is actually in control of the client and the client uses usually a fixed SMTP server with fixed TLS settings, so no problem with MX spoofing and connection downgrades. If the client is instead configured to use TLS only if available the attack might work against implicit TLS too.

5.) Can't you configure the server/MTA to drop connection (bounce email) if other MTA isn't TLS compatible i.e. switch from opportunistic to required, or is that something you can only do on the client (like Thunderbird) for client-server connection?

Most servers can be configured this way but most servers have to talk to a variety of servers and they don't know up front if the server will support TLS. But sendmail for example can be configured to talk only TLS to specific servers or not to talk TLS to specific other servers. So this configuration can be hardened to known good servers, but this must be done manually.

6.) If #5 is possible and email bounces, will sender receive a 'failed to send' notice?

Yes, usually this is the case, like with all delivery errors. But like most of the other delivery errors only technically fit people might understand these error messages.

7.) As far as certificates not getting checked, is that something that can be fixed with proper configuration/implementation. I read that one of the issues was that a lot of certificates are self signed and most MTA's accept them by default so they can stay compatible with more MTA's. Is that part of what you were referring to?

It depends on the server. Lat time I looked postfix was fine but sendmail could not properly check the hostname against the certificate. And yes, self-signed is one problem and the other main problem are missing intermediate certificates. But since mail delivery works anyway (since certificate errors are ignored by the sender) most admins don't realize the bad configuration or don't care.

8.) Lastly, say all non-trivial fixes were put in place, and i understand the other MTA/client is out of my control. Can these MITM and injection attacks target specific connections going to and from my email address? In this situation they haven't broken into my server and they have no previous experience with my contacts.

Again, this is only hop-by-hop encryption and the mail is available in clear for reading and modifying on each of the hops. Thus an attacker on any of the hops involved in the transfer (usually at least two, one at the sender and one at the recipients site) can intercept and also modify the mails and can of course restrict itself to deal only with selected mails. The only protection is end-to-end encryption using PGP or S/MIME.