multiple physical interfaces with IPs on the same subnet

I have a linux box with 9 NICs on it, and I want eight of them to have unique addresses on the same subnet, e.g.:

ifconfig eth1 192.168.123.1 netmask 255.255.0.0
ifconfig eth2 192.168.123.2 netmask 255.255.0.0
ifconfig eth3 192.168.123.3 netmask 255.255.0.0
...
ifconfig eth8 192.168.123.8 netmask 255.255.0.0

The default ARP behaviour is extremely counterproductive in this case, since it results in all traffic for all IPs passing exclusively through eth1, which is pretty much the exact opposite of what I want.

So I rummaged around and ended up making some changes to sysctl such as this:

net.ipv4.conf.all.arp_filter=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

That did prevent eth1 from impersonating all the others, but I still can't ping anything other than eth1's address successfully. (e.g. from a second computer on the same switch, only 192.168.123.1 responds to ping)

I'm guessing that I need to do something with arptables or iproute or SOMETHING, but I'm lost at sea in this field.

Bonus Points: Solution must be compatible with Linux 2.6.27.27. (More specifically, Slax 6.1.2)

Solution 1:

You need a strong end system model. Linux is fundamentally built around a weak send system model, so it's really not a good OS choice for this application.

You will have to fake every piece of the behavior you need, from ARP to policy routing to source address selection. You will also need filters to prevent packets from being accepted if they arrive on the wrong interface.

The definitely necessary steps are:

1. Configure arp_filter=1 and arp_ignore=2 on all interfaces.

2. Add per-interface, source-based routing for outgoing traffic. (Destination interface must be chosen based on source address.)

3. Add per-interface ingress filtering to silently drop packets received on the wrong interface. (Packets with a destination address assigned to another interface.)

Unfortunately, there is no consensus on whether these three steps are all that is needed. The weak end system model is built into the entire Linux TCP/IP stack, and it's not clear what might go wrong with subtle issues like multicast.

It's not clear how you would choose the output interface for broadcasts, for example. Should it go out all of them? Maybe. What is the right behavior if the stack gets an outbound broadcast with a source address not assigned to one of the interfaces?

Again, you have chosen the wrong tool for the job.

Solution 2:

You more likely want to create a bridge with the 8/9 interfaces and then assign an IP address to that bridge (bridge-utils packet, command 'brctl add').

This way the bridge will act like as a switch and can have an IP address into your subnet.

Solution 3:

I would recommend bonding the physical interfaces then configure all of the addresses on the single bonded interface.

You'll need support on the switch as well.

Here's a mini tutorial that you can use to get started.