What problems arise when we use a self-signed certificate for the SMTP protocol?

On server to server communications there's no problem, because SMTP servers accepts any certificate and even falls back to unencrypted connection. In that sense, there's no public key infrastructure for MX servers. Against downgrade attacks there's DNS-Based Authentication of Named Entities (DANE) (see RFC 7672 on SMTP Security via Opportunistic DANE TLS), utilizing public keys published in DNS over public keys signed by a CA. Self-signed certificates are ok for that.

For client to server communications it's at least more convenient to have a trusted CA signed certificate. If users learn to accept an exception, it's way more likely they do so even when there's a MitM attack in place. Moreover, e.g. Android mail client only allows requiring a trusted certificate or accepting any, making the MitM attack even easier. That's why I would recommend using a CA signed certificate for client communications.

It's even possible to get a free Let's Encrypt certificate for your SMTP/IMAP server as it's not limited to use the same certificate for multiple protocols. This pretty much removes any need for self-signed certificates with publicly-referenced mail servers. (Example configuration for Postfix.)

A self signed certificate might deter trivial attempts to intercept mail, but would be of limited value as they can be MITM'd. If you do want to go this route you should create a CA and have your users accept the CA. Unfortunately this is possibly a bad idea for your users (as you could sign rogue certs), and of limited value because the traffic could still be intercepted between other MTAs and yours.

Something no one else has commented on - while encrypted delivery is not part of the SMTP protocol, some providers (eg GMAIL) will mark the email as having been encrypted or not - and I speculate (I don't use gmail, so don't know for sure) that MTAs using self signed certs won't be considered as secure.

As others have mentioned, using LetsEncrypt is generally not to big an ask, and a better solution.