Infiniband on Linux?

The Infiniband itself is not a big deal to set up. You have to load the driver for your device and if you don't want to use anything else then IP the ipoib kernel module.

1 IP OVER INFINIBAND 2
3 The ib_ipoib driver is an implementation of the IP over InfiniBand 4 protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib 5 working group. It is a "native" implementation in the sense of 6 setting the interface type to ARPHRD_INFINIBAND and the hardware 7 address length to 20 (earlier proprietary implementations 8 masqueraded to the kernel as ethernet interfaces).

After this you are able to use your device as a normal ethernet device.

Do you absolutely need Inifiband? Or do you just need high bandwidth links between things? You might be better off bonding several 1GigE links or going with 10GigE.

Easy? cheap? not really, but definitely possible. Since you're not going for the budget solution (that's six figures for the metal only), you'd probably want to use support from a paid, enterprise, OS.