Specifying a preferred route when there are multiple links to same network

Solution 1:

There are some things to take care about when you're multi-homing hosts. First thing is that you need to be aware of the way the Linux TCP/IP stack will handle multiple interfaces being in the same subnet regarding ARP queries and answers - this setting is the interface's arp_filter value that you can query via sysctl(1) or the /proc file system.

0 - (default) The TCP/IP stack will respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces.

1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface. In other words it allows control of which NICs will respond to an ARP request and will finally let your TCP/IP flows run.

You should first enable arp_filter on all the interfaces that are in the same subnet and you could easily add an entry to your routing table for your iSCSI portal to use a specific iface and adjust the other interfaces metric to so that one is prefere one over the other.

Another option is to setup source based routing as the default destination based routing will act exactly as you described wrt all ifaces being in the same subnet.

The reason why eth1 is taken out is that it's IP has the lowest numerical value and is such chosen for communication with that network.

Solution 2:

To solve my problem I wrote nethook, a daemon that runs scripts when network interfaces change state on RHEL-based distributions. I have it run this script for interfaces whose route's metric I want to increase.

EDIT: I wrote nethook before I was aware of ifup-local and ifdown-local. You can probably use them instead.