What host name should the SSL certificate for an SMTP server contain?

This is actually not explicitly defined anywhere, and whether or not the server should be "trusted" depends on the client (which could of course be another mail server) connecting to it; quoting from the relevant RFC (RFC 2487):

If the SMTP client decides that the level of authentication or
privacy is not high enough for it to continue, it SHOULD issue an
SMTP QUIT command immediately after the TLS negotiation is complete.
If the SMTP server decides that the level of authentication or
privacy is not high enough for it to continue, it SHOULD reply to
every SMTP command from the client (other than a QUIT command) with
the 554 reply code (with a possible text string such as "Command
refused due to lack of security").

The decision of whether or not to believe the authenticity of the
other party in a TLS negotiation is a local matter. However, some
general rules for the decisions are:

- A SMTP client would probably only want to authenticate an SMTP
  server whose server certificate has a domain name that is the
  domain name that the client thought it was connecting to.

What this basically means is, when the server offers TLS encryption using a given certificate, the decision about accepting or refusing it depends entirely on the other part, which will probably want the name on the certificate to be the same it connected to, but could very well accept it even if it doesn't match.

But wait, there's more. Quoting again from the same RFC:

Upon completion of the TLS handshake, the SMTP protocol is reset to
the initial state (the state in SMTP after a server issues a 220
service ready greeting). The server MUST discard any knowledge
obtained from the client, such as the argument to the EHLO command,
which was not obtained from the TLS negotiation itself. The client
MUST discard any knowledge obtained from the server, such as the list
of SMTP service extensions, which was not obtained from the TLS
negotiation itself. The client SHOULD send an EHLO command as the
first command after a successful TLS negotiation.

So, what the server is saying in response to HELO/EHLO before the TLS handshake doesn't seem to actually matter at all.

In my experience, self-signed certificates work pretty well on Internet-facing mail servers, which means the other mail servers aren't even bothering to validate them, they'll just happily accept anything that can provide TLS encryption, regardless of the issuing authority or subject name.

An MTA delivering mail to your domain is going to lookup the MX record (which will yield a hostname), and then lookup an A record for that hostname. The hostname which it is connecting to is therefore the MX hostname, and so that is what will be verified against the SSL certificate common name. Verifying the HELO hostname doesn't make sense because the server can provide any HELO hostname it wants -- it doesn't provide additional security.

That said, strictly verifying SSL certificates when delivering mail isn't particularly useful at the moment, since MTAs will (almost always) fallback to non-SSL delivery, since that's how SMTP works at the moment. The sensible configuration is therefore to use SSL if the MX server offers it, regardless of whether the SSL certificate verifies or not (since encryption without authentication is better than no encryption and no authentication). You therefore might as well use a self-signed certificate for this purpose.

The task of verifying the server certificate and that it matches the host name of the server is purely the client's role, for any protocol using SSL/TLS.

As such, the host name in the certificate should match the name that the client is trying to access.

When the SSL/TLS connection is initiated up front (SMTPS), the server has no way of seeing what the HELO message says before the connection is established, so it must use the one with which it made the request.

When using SSL/TLS after STARTTLS, the client still intends to talk to the server with which it was configured, so that's still what it should check. Failing that would make MITM attacks possible:

• C->S: Hello, I'm Alice, I want to talk to Bob.
• S->C: Hi, I'm Chuck, here is my cert for Chuck.
• C->S: Oh yes, your cert is indeed valid for Chuck. Let's proceed.
• ... There's a flaw there of course, since Alice wanted a secure communication with Bob.

In both cases, it's the MX address that should be used.

The host name matching rules have recently been gathered across protocols in RFC 6125, but few clients implement it fully (it's more of a best practice RFC than a complete change, and it's still quite recent).

In its appendix, it summarises what existed about SMTP before (taken from RFC 3207 and RFC 4954). In particular "The client MUST NOT use any form of the server hostname derived from an insecure remote source (e.g., insecure DNS lookup)." (which applies to the server's banner of course). Apart from this, the SMTP legacy rules were a bit more relaxed than HTTPS regarding Subject Alternative Names (should instead of must be used).

The modern way is definitely to put the host name in a Subject Alternative Name DNS entry. Wildcard usage is also discouraged.