Infiniband OpenSM N-to-N port routing configuration
https://community.mellanox.com/s/question/0D51T00006RVtlU/rdmacm-connection-setup-issues
For the record, the issue has been largely resolved by ensuring the ibacm service (Infiniband Assistant Communication Manager) was running on all servers.
Running sudo ibacm on all servers solved the issue, don't ask me why...