Why can't the GPUs communicate in a multi-GPU server?
While there is no explicit "IOMMU off" setting in this BIOS flavour, the problem is still with the BIOS configuration.
In the BIOS, go to "Integrated Devices" and change the "Memory Mapped I/O Base" setting from the default "56TB" to "12TB". This will solve the issue. There is no need to add any extra kernel parameters.