Is it good practice or too draconian to reject mails from mailservers with no RDNS

Solution 1:

I have tried multiple approaches with the HELO/EHLO checking with a fairly decent sized customer base of between 100k-200k users and ended up going with a solution that does the following:

• Check that the HELO/EHLO contains a domain name. -- This mostly boils down to does the name have a dot in it. Checking the DNS on the name led to MANY failed servers as its not uncommon to have the server present an internal name or something you can't resolve properly.
• Check that the IP has a reverse DNS record. -- Again this is a lax setting as we don't check it against the HELO/EHLO. Checking against the HELO/EHLO created so many tickets this setting didn't last even a single day.
• Check the senders domain name is valid. -- This is a basic check to make sure if we do have to bounce the message there will at least be some way to find a server for it.

Here's the Postfix block we use for these checks:

smtpd_recipient_restrictions =
    reject_non_fqdn_sender,
    reject_unauth_destination,
    reject_unknown_reverse_client_hostname,
    reject_invalid_helo_hostname,
    reject_non_fqdn_helo_hostname,
    reject_unknown_sender_domain,
    reject_non_fqdn_recipient

Solution 2:

It's extremely common to block SMTP servers that don't have these basics:

1. Hostname in HELO forward resolves to IP connection originated from.
2. Connections origin IP reverses to the Hostname claimed in HELO.
3. If SPF, DKIM, or DMARC policies exist, verify.

Anyone griping about getting blocked because of one of these should be tarred and feathered.
People who end up getting blocked for other reasons, particularly situations that rely on RFC conformance in "abnormal" situations, I'll have sympathy for. Spam is such a problem that there's just no excuse for missing the basics though.

Solution 3:

I would expect sending MTA to have valid RDNS but insisting on matching EHLO would depend on who 'the customers' are. You can find some interesting guidelines in RFC5321:

2.3.5.

(...) The domain name given in the EHLO command MUST be either a primary host name (a domain name that resolves to an address RR) or, if the host has no name, an address literal (...)

4.1.4.

(...) An SMTP server MAY verify that the domain name argument in the EHLO command actually corresponds to the IP address of the client. However, if the verification fails, the server MUST NOT refuse to accept a message on that basis.

but then in 7.9.

It is a well-established principle that an SMTP server may refuse to accept mail for any operational or technical reason that makes sense to the site providing the server.(...)

Solution 4:

Reverse lookup not necessarily points to the hostname provided in HELO. Sometimes multiple domains hosted on the same host, and all of them have the same IP address. But when you try to do the reverse lookup, you'll get the name that has been placed in the PTR-record. It's obvious that both FQDNs will be different - and that is completely acceptible.

The only circumstance that allow to drop message is failed reverse lookup. Any succesful lookup means that host is valid. Names shouldn't match.

Solution 5:

I'm wondering whether I should also block hosts that don't have a valid RDNS matching the EHLO?

No, you shouldn't. Blocks a whole email only by one criteria it's a bad practice.

If I do this, am I going to make trouble for much legitimate mail and upset my customers?

more likely you do and will lost legitimate mail

I'm also wondering if I can compromise by checking that RDNS is at least set to something, but not try to match it to the EHLO. Is this possible with Postfix (and is it useful)?

yes, it possible. You can use reject_unknown_reverse_client_hostname instead of reject_unknown_client_hostname

Unfortunately, postfix doesn't have a flexible options for "complex decision". In exim you can add some points for such mails, for e.g.

Score = 0 
1. The HELO or EHLO hostname is not in fully-qualified domain or address literal form. Score +=10
2. The HELO or EHLO hostname has no DNS A or MX record. Score +=20
3. The HELO or EHLO hostname is listed with the A record "d.d.d.d" under rbl_domain. Score +=20
4. The sender domain has no DNS A or MX record. Score +=10
5. SPF checks return softfail. Score +=10, fail, Score +=20
...

And so on. After all checks will be completed and if you had Score > 100, you can reject mail. Actually you can get such behavior, but you would need to write your own policy service