infiniband network between 3 servers [closed]

Let's say I have 3 different servers, each one with an infiniband card. Each card has 2 different ports. (I don't know about the model yet)

Is it possible to create 3 different networks and to allow the 3 servers to communicate with each other without any problems? (and any spof).

I guess I just have to setup the /etc/hosts correctly.

I really don't know about infiniband, so please help me :)

Thanks in advance.

EDIT: Point is to NOT USE a switch!

Solution 1:

You're just talking about doing a point-to-point connection from each host to each of the other two right?

I've not tried it, but from what I have heard you can do point-to-point connections using IB without using a switch (just as you can with ethernet).

Problems might come in with the subnet manager(s) - I'm not sure if each node's subnet manager can run actively for one of the connected networks and backup for the other. You may need to run two subnet managers per box and bind each to a specific port. Not sure how to do that either but definitely worth researching that side of it more.

And obviously having a couple of switches would make things easier (and more scalable).

Solution 2:

Yes, just buy one or two (for resilience) IB switches. That said you don't mention which protocols you want to use, though it seems like you might want to run IP over IB, which works - and editing hosts (or having a proper DNS environment) will help to make that work but IB is really designed for IB-specific protocols, IP over IB works great, it's not just a mainstream usage case ok.

Solution 3:

Your question basically boils down to this:

I have 3 servers with Inifiniband. Can I create a network between them?

Which is analogous to

I have 3 servers with Inifiniband Ethernet. Can I create a network between them?

So the answer is yes. But then when you throw this into the mix:

Point is to NOT USE a switch!

The answer is no. You need a switch to get 2 or more of anything to work together. Even if you daisy chain them off each other (eww) then you still have a switch in the middle of the daisy chain. I'm not saying you can daisy chain them though, so don't take that as an assumption that you can.