Ubuntu Linux - multiple NICs, same LAN... ARP responses always go out a single NIC
We've got AT&T U-Verse internet service, which has an extremely boneheaded DSL gateway.
We have 5 IPs (netmask 248), but the gateway is incapable of doing anything other than a single IP -> single MAC address mapping.
We have a single firewall machine, and we redirect different IP/port combos to different places inside a DMZ.
Our solution so far is to have a VMWare virtual machine on the firewall with 4 additional NICs in it, to get the other 4 IP addresses... however we have a problem.
The gateway is basically doing an ARP ping to see if the IP is responding on the expected MAC. With 4 NICs all on the same LAN, linux is responding to ARP requests for ALL IPs using a single interface. That's not what the gateway is expecting, and it's messing up the 3 other NICs. The gateway refuses to route incoming traffic for IPs where the ARP ping results aren't the expected MAC.
How can we get the ARP replies for eth0's IP to go out eth0, eth1's IP to go out eth1, etc?
EDIT
Christopher Cashell's response does not work in this situation. I had great hopes reading it, but... nope.
EDIT 2
Solved! See my answer below.
Your chosen solution works, but there are alternatives that don't involve arptables. (Christopher Cashell was on the right track, originally, but he was off by a smidge.)
In short, you want to set these parameters:
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
These should be available when running a modern, 2.6 series Linux kernel. Check and make sure you that '/proc/sys/net/ipv4/conf//arp_announce' and /proc/sys/net/ipv4/conf//arp_ignore' exist on your system.
The 'arp_filter' parameter only works when your various IP addresses share a LAN segment but use different IP subnet. If they share the IP subnet, too, you need to use 'arp_ignore' and 'arp_announce', as above.
(I believe you may also need to set 'arp_filter' back to '0', too.)
Okay, here's the solution. First, a recap:
Here's my basic network plan:
eth0 10.10.10.2 netmask 255.255.255.248
eth1 10.10.10.3 netmask 255.255.255.248
eth2 10.10.10.4 netmask 255.255.255.248
eth3 10.10.10.5 netmask 255.255.255.248
All the interfaces overlap. This is technically wrong, and the source of all my woes... but I have to do it because of this dumb residential gateway.
First, broadcast ARP requests go to all of these. Since all 4 IPs are valid local addresses, all 4 interfaces are going to try to respond.
1) install arptables. Add this someplace during boot (/etc/rc.local here):
arptables -F INPUT
arptables -A INPUT -i eth0 --destination-ip ! 10.10.10.2 -j DROP
arptables -A INPUT -i eth1 --destination-ip ! 10.10.10.3 -j DROP
arptables -A INPUT -i eth2 --destination-ip ! 10.10.10.4 -j DROP
arptables -A INPUT -i eth3 --destination-ip ! 10.10.10.5 -j DROP
This will prevent broadcasts from going into the wrong interface. So, the correct interface will now be the only responder.
That by itself is not enough. The next bit is an ARP table problem. The requesting PC probably already has an ARP table entry, and so Linux it's going to use the interface associated with that. Until that ARP table entry expires, it's going to be trying to send ARP responses using that entry's interface, not the one associated with the ARP request.
The sysctl option rp_filter appears to be rejecting outgoing ARP response packets if they're on the wrong interface. So...
2) Disable rp_filter.
On Debian/Ubuntu, this means commenting out the two rp_filter lines in /etc/sysctl.d/10-network-security.conf.
This option was enabled for a reason... namely to help prevent cross-interface spoofing attacks. I read it verifies that the packet is legal for the interface it is coming in or going out on (by swapping MACs and IPs and seeing if it still routes over the same interface). So, normally it would be a bad idea to turn it off. In my case, all interfaces are on the same network... so that check doesn't matter really at all.
If I do add another interface and need the spoofing protection, perhaps can craft some arptables/iptables entries to do the same thing.
This has to do with the way Linux handles IPs and NICs. Basically, it treats an IP address as if it belongs to the box, and not just the specific NIC. The result is that you can get ARP responses from IP addresses on interfaces you wouldn't expect.
The solution is a sysctl option. As I recall, what you're looking for is:
net.ipv4.conf.default.arp_filter=1
net.ipv4.conf.all.arp_filter=1
That will fix the problem for you. Just add those to /etc/sysctl.conf and run 'sysctl -p' (or run each line as the argument to 'sysctl -w'.
This will cause Linux to only respond to ARP requests on the interface that an IP address is actually assigned to.
The accepted answer in combination with this: http://www.linuxquestions.org/questions/linux-networking-3/multiple-interfaces-all-traffic-flows-through-just-one-538701/
where static routes are used to communicate with only desired IPs over certain interfaces, created a powerful and simple solution to a similar problem I had.