Simple Multihomed Linux Server Issue

I have a Linux server running CentOS 6.4 that is used as an iSCSI target. The server is multi-homed with two NICs, both on the same subnet. iSCSI multipathing takes care of the load-balancing/failover, so all I need is for each NIC to operate independently. How do I configure this system to avoid all the weird routing and ARP issues that always come along with a multi-homed setup, such as traffic being returned on a different interface from the source, or one interface accepting traffic sent to the IP of the other. The bonding solution is not an option, as it has issues with the iSCSI connection.

The important stuff:

• eth0: IP 10.1.1.242 / SN 255.255.252.0 / GW 10.1.1.254
• eth1: IP 10.1.1.243 / SN 255.255.252.0 / GW 10.1.1.254

I realized I never followed up to this question. Using some excellent internet resources here and here, I came up with the following configuration. Hopefully this helps someone out.

Assume you have two interface, eth0 and eth1, with IP addresses of 10.1.1.242 and 10.1.1.243. This is all on a /22 network with a default gateway of 10.1.1.254.

• First create two routing tables, one for each NIC:

  echo "1 lan1" >> /etc/iproute2/rt_tables`
  echo "2 lan2" >> /etc/iproute2/rt_tables`
  

• Next, add routes for each interface to the appropriate tables:

  ip route add 10.1.0.0/22 dev eth0 src 10.1.1.242 table lan1
  ip route add default via 10.1.1.254 dev eth0 table lan1
  ip route add 10.1.0.0/22 dev eth1 src 10.1.1.243 table lan2
  ip route add default via 10.1.1.254 dev eth1 table lan2
  

• Finally, add rules to determine which table is used:

  ip rule add from 10.1.1.242/32 table lan1
  ip rule add to 10.1.1.242/32 table lan1
  ip rule add from 10.1.1.243/32 table lan2
  ip rule add to 10.1.1.243/32 table lan2
  

This should keep traffic from crossing NICs internally, allowing you to preserve redundancy or use each NIC for different functionality.