Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $ on an expensive switch.

Do I need a special cable or do the standards one work ok? On standard ethernet I would need a cross connect cable.

Solution 1:

Yes, you can directly connect them. But you must ensure that on at least one of the machines you run the opensm subnet manager.

From the software side, if you were using Ubuntu 14.04, install these packages:

sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev

Then add these to /etc/modules

mlx4_ib
ip_ipoib
rdma_ucm

Add an IP address to the card. edit /etc/network/interfaces

auto ib0
iface ib0 inet static
  address 10.0.0.1
  netmask 255.255.255.0

Reboot.

Opensm will automatically start. Repeat the above for the other machine but use a different IP: 10.0.0.2.

All going well you should be able to ping 10.0.0.2 from 10.0.0.1 and vice versa.

Run some tests:

sudo ibnodes

Oh, there is a bug in scripts for ibnodes and several others. You can fix them by editing them. Mostly they are shell scripts that point to /usr/local/sbin when they are actually calling apps in /usr/sbin.

If you find one, go right ahead and edit it and change IBPATH accordingly. I have never gotten around to filing a bug report against Ubuntu for that. Hopefully someone else will do it. It's been a problem for ages.

Output will look something like:

Ca  : 0x001a4bffff0c9374 ports 2 "HP Lion Cub DDR 128MB"
Ca  : 0x001a4bffff0c446c ports 2 "HP Lion Cub DDR 128MB"
Ca  : 0x001a4bffff0c4438 ports 2 "HP Lion Cub DDR 128MB"
Switch  : 0x000b8cffff006aa8 ports 24 "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 2 lmc 0