Is it ok to use self-signed certificates for smtp transport?

I ran an MTA with a self-signed certificate for a couple of years, until real ones got cheap enough that I could no longer be bothered to do so, and I didn't have a single rejection because of the unsigned certificate in all that time. I never had a single complaint about a mail being marked as spam because of it, either; if anything, using TLS often seems to mark you out as a non-spamming professional.

In my opinion, it is definitely worth enabling SMTP TLS if you can, whether or not you pay for a third-party-signed certificate.

Edit in response to your comment: It's not that someone couldn't choose to restrict inbound mail on that basis; I've never come across it, is all. A third-party-signed certificate is still useful to prove there is no man-in-the-middle attack happening; but that doesn't seem to be a serious problem in the MTA world at this time. If that starts to change, we could well find that people start to insist on signed certificates.

Security exists to address threats, so if the threat model changes, the range of sane and proportionate security responses will change with it.

Like MadHatter said, at the moment encryption using a self-signed certificate is actually a step up in comparison to many other small time SMTPs if you are talking SMTP-relaying.

There are a few drawbacks, however, but not of the kind you expect. The most important one being your SMTP-clients choking on encryption. There are many small SMTP-clients deployed in off-the-shelf solutions for common business problems that don't like SMTP encryption and fail -- often silently! So, if you are using standard software with built-in mail clients, check them before you switch.

Other than that I'd say, if you are concernced about your mail being marked as spam, consider setting up SPF and DKIM. That helps very much.

The question is certainly somehow dated, but earns a revisit and should receive an updated answer.

Regarding the impact on outbound delivery and spam filtering: According to Google's transparency report on e-mail encryption in transit to/from Google, about 90% of outbound and 93% of inbound e-mails are transmitted using TLS. So with some respect, TLS for e-mail is becoming the norm and considering the high volume of spam, just using TLS as another scoring value in a spam filter is not of much value. As today inbound TLS certificates are usually not being used to authenticate outbound connections, the certificate's contents (metadata, who signed the cert,…) don't impact spam filters on outbound e-mail at all.

Historically, MTAs do have limited options to handle any certificate errors: reject sending the message at all or revert to send the message via plaintext connections. Rejecting the message will result in user support inquiries while reverting to plaintext doesn't provide security benefits over using at least "some" level of encryption. Accordingly, most MTAs have been using an "opportunistic TLS" configuration: prefer using TLS if available, regardless of the exact certificate to validate.

However, there has been some slight shift visible in the industry over the past decade.

Since 2010, the German Association of the Automotive Industry has been recommending using at least opportunistic TLS for e-mail and recommends mandatory TLS for confidential e-mail.

Some e-mail service providers also started introducing optional mandatory outbound TLS, for example German e-mail provider posteo.de did so in 2016.

Posteo's feature can be enabled per user, and if the message cannot be transmitted using an encrypted connection, the sending user will receive an error message. The user then can decide to either (temporarily) disable that feature (e.g. to ask the recipient on upgrading their mail service). Within 2 weeks after the feature has been released, quite a few e-mail services started supporting (opportunistic) TLS.

So not offering TLS at all and just relying on unencrypted mail may result in partial inbound issues with some senders. However, most of those cases at that time did seem to be fine with self-signed certificates.

A research paper on E-Mail security from 2015 discovered 64% of domains do use STARTTLS-supporting mail servers whose certificate did validate with Mozilla's NSS root store - so 36% have been using self-signed, invalid or expired certificates.

More recently, MTA strict transport security did improve the overall trend: if domains hosted on your mail server do make use of MTA strict transport security (RFC 8461), those domains have also agreed to enforce inbound mail via TLS using trusted certificates. RFC 8461's section 4.2 does clearly tell

The certificate presented by the receiving MTA MUST not be expired and MUST chain to a root CA that is trusted by the Sending MTA.

Of course, this only applies to domains using MTA strict transport security.

So after all: As of today, issues with outbound e-mail are unlikely. Regarding inbound e-mail, you're highly encouraged to use proper TLS certificates issued by a public CA, as not doing so may result in some delivery issues. Some of those might be addressable within your domain (like MTA-STS), but others may not be (like sending servers enforcing mandatory outbound TLS).