iptables only allow smtp connections with start tls

Is it somehow possible to block/abort unencrypted smtp connections with iptables?

effectively rejecting all connections which dont do a start tls.

This is unavoidably tricky. The nature of TLS is such that a plaintext connection to your MTA has to be established before TLS can be negotiated, so iptables (operating as it does at the transport layer) is ill-designed to influence issues at the application layer.

You could write another target module and direct your traffic through that, but unless you're a networking God, this is probably no more feasible for you than it is for me. And I definitely don't know how to do it.

The upshot is that application-layer stuff is much easier to enforce inside the application. You don't say what MTA you're using, but I suspect that most MTAs that are bright enough to do TLS are bright enough to mandate it.

I use sendmail. There's a nice piece on mandating TLS from various providers at http://www.brandonhutchinson.com/Using_TLS_with_Sendmail.html , which directs me to the access database entry

TLS_Clt:communication_partner_MTA                           PERM+VERIFY:112

which requires a particular communication partner, presumably identified by IP address, to both authenticate with a key of at least 112 bits length, and have a properly-signed certificate. The sendmail config page at http://www.sendmail.org/documentation/configurationReadme , in the ANTI-SPAM CONFIGURATION CONTROL section, says that access db entries involving IPv4 addresses can take the form of a single octet, which then apply to all addresses beginning with that octet. So I speculate, and I stress it's just speculation, that sendmail would allow me to have a series of entries

TLS_Clt:1       PERM:112
TLS_Clt:2       PERM:112
TLS_Clt:3       PERM:112
....
TLS_Clt:223       PERM:112

Mandating encryption (though not verifiably-signed certificates; self-signed TLS certs are very common, and I'd be inclined not to bar them) from all IP addresses. I would also not have an entry for TLS_Clt:127, as localhost should probably not be so restricted.

I repeat that I've not tested any of the above, and if your MTA is something other than sendmail, the above won't be specifically helpful; but I wanted to show that my MTA (at least) seems to have hooks for doing what you want. Good luck with your investigations.

Iptables is mostly designed to filter on the metadata of a network packet, such as IP headers, TCP options, and so forth. TLS is handled at the application level: you need to watch what happens inside the TCP stream.

You can inspect the contents of packets with the string extension, and you can write userland filters if the kernel ones aren't enough. But that would be very hard: for example, TCP streams can be broken into packets in arbitrary ways, and the TCP packets can be received out of sequence... You'd have to deal with all these issues (duplicating the work of the kernel).

What you're looking for here is an SMTP proxy that forwards TLS connections and blocks non-TLS connections. Iptable's contribution would be only to ensure that SMTP traffic does go through the proxy.