How can I decrypt STARTTLS communication over SMTP in a packet capture (if I have the private key)?

Knowledge of the RSA private key is only sufficient if RSA key exchange is done. Modern systems instead use Diffie Helmann key exchange (DHE*, ECDHE* ciphers) to provide forward secrecy. In this case the private key is only used in authenticating the server but not for the key exchange and thus knowledge of the private key does not help in getting the encryption keys.

Instead the SSL pre master secret would be needed since this is used to derive the encryption keys. While some browsers support exporting this secret in some special debug modes other clients usually don't.

As for your specific problem I would instead to try a man in the middle "attack" which logs all traffic in plain even though the client gets encrypted traffic. This works similar to man in the middle proxies for HTTPS, only that it supports the initial plain connection and understands that STARTTLS upgrades to TLS. From a quick search I've found starttls-mitm and that sslsplit has some beta support for this. Of course the client needs to connect to the man in the middle proxy and trust the certificate it gets, which might be the original server certificate since you seem to have access to this.

Another option would be to enable debugging in the specific MTA in case this provides detailed enough information for your specific problem.

It may be that the wireshark SMTP protocol parser doesn't know how to handle TLS/SSL. But maybe you can use the HTTP protocol parser instead. Go to edit->preferences->protocols->HTTP and add the port to SSL/TLS ports