Ghost ping on a multi-NIC Linux system

It's not only about routing (layer 3) but also about ARP (linking layer 2 and 3). Linux will typically answer all ARP requests with the same MAC, that means peers will have only one MAC address in their ARP cache for the 3 IPs and will reach only one card. It's possible that sometimes the elected MAC will change, causing all kind of troubles.

There's an available feature to prevent this. That's the arp_filter setting, but it also requires a few routing rules (per ip and/or per interface). Using only arp_filter or only routing tables might give worse results than not using them at all. Both are needed together. Because OP didn't show what routing policies were used, I put a complete solution below, considering no special routing policies were in place. Values for routing tables (100, 101, 102) were chosen at random.

• LAN

  • try before:

    ip route get 192.168.1.1 from 192.168.1.10
    ip route get 192.168.1.1 from 192.168.1.11
    ip route get 192.168.1.1 from 192.168.1.12
    

    Note the results: all are using the same card (that doesn't mean it's always what happens anyway).
    On a peer server in the same LAN, as fast as possible:

    ping -c1 192.168.1.10 &
    ping -c1 192.168.1.11 &
    ping -c1 192.168.1.12
    ip neigh show to 192.168.1.10
    ip neigh show to 192.168.1.11
    ip neigh show to 192.168.1.12
    

    most certainly all 3 IPs resolve to the same MAC, thus card: the only card that will receive traffic from this peer.

  • alter the settings (Warning: risk of connectivity loss):

    echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_filter
    echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_filter
    echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_filter
    
    ip route add 192.168.1.0/24 dev eth0 table 100
    ip route add 192.168.1.0/24 dev eth1 table 101
    ip route add 192.168.1.0/24 dev eth2 table 102
    
    ip rule add from 192.168.1.10 lookup 100
    ip rule add from 192.168.1.11 lookup 101
    ip rule add from 192.168.1.12 lookup 102
    

  • flush peers' ARP caches (with DAD requests doubling as GARP) to speed up recovery if needed. Not needed at boot time since there were no such caches. Use iputils' arping, not the standalone arping tool (RHEL: iputils, Debian: iputils-arping, not the arping package), because syntax differs.

    arping -c5 -s 192.168.1.10 -D 192.168.1.10 -I eth0 &
    arping -c5 -s 192.168.1.11 -D 192.168.1.11 -I eth1 &
    arping -c5 -s 192.168.1.12 -D 192.168.1.12 -I eth2 &
    

  • redo the previous checks and compare: everything should be as expected, and neighbour tables on peer servers should show one different MAC per IP, matching the correct card's MAC.

• WAN

  The default route will still use the "default" settings from table main, most likely using eth0. This route has to be duplicated on each table. Assuming the default router is 192.168.1.1 (else, adjust) that would be:

  ip route add default via 192.168.1.1 dev eth0 table 100
  ip route add default via 192.168.1.1 dev eth1 table 101
  ip route add default via 192.168.1.1 dev eth2 table 102
  

Now if a cable is disconnected, the expected result will happen: the IP set on the corresponding NIC will become unavailable 100% of the time.