Force local IP traffic to an external interface

Solution 1:

I expanded on caladona's answer since I could not see response packets. For this example:

1. On my local PC I have NIC's on different subnets, 192.168.1/24, 192.168.2/24
   
2. There is an external router/PC that has access to both subnets.
3. I want to send bi-directional traffic over the NICs on the local PC.
4. The configuration requires two unused IP addresses for each subnet.

Local PC iptable routes are set to SNAT and DNAT outgoing traffic to the 'fake' IP.

iptables -t nat -A POSTROUTING -d 192.168.1.100 -s 192.168.2.0/24 -j SNAT --to-source      192.168.2.100
iptables -t nat -A PREROUTING  -d 192.168.1.100 -i eth0           -j DNAT --to-destination 192.168.1.1
iptables -t nat -A POSTROUTING -d 192.168.2.100 -s 192.168.1.0/24 -j SNAT --to-source      192.168.1.100
iptables -t nat -A PREROUTING  -d 192.168.2.100 -i eth1           -j DNAT --to-destination 192.168.2.1

The rules do the following:

1. Rewrite 192.168.2.1 source to 192.168.2.100 on outgoing packets
2. Rewrite 192.168.1.100 destination to 192.168.1.1 on incoming packets
3. Rewrite 192.168.1.1 source to 192.168.1.100 on outgoing packets
4. Rewrite 192.168.2.100 destination to 192.168.2.1 on incoming packets

To summarize, the local system now can talk to a 'virtual' machine with addresses 192.168.1.100 and 192.168.2.100.

Next you have to force your local PC to use the external router to reach your fake IP. You do this by creating a direct route to the IP's through via the router. You want to make sure that you force the packets onto the opposite of the destination subnet.

ip route 192.168.1.100 via $ROUTER_2_SUBNET_IP 
ip route 192.168.2.100 via $ROUTER_1_SUBNET_IP

Finally to make this all work, the external router needs to know how to reach the faked IPs on your local PC. You can do thins by turning on proxy ARPs on for your system.

echo 1 | sudo tee /proc/sys/net/ipv4/conf/all/proxy_arp
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

With this setup, you can now treat the fake IPs as a real system on your local PC. Sending data to .1 subnet will force packets out the .2 interface. Sending data to the .2 subnet will force packets out the .1 interface.

ping 192.168.1.100
ping 192.168.2.100

Solution 2:

I successfully used the following on Linux to test throughput on a new dual-port 10Gbps card in "loopback" mode, that is, one port plugged directly into the other. This is all just a bit of voodoo just to force packets out the wire, but if you don't, Linux will just short-circuit the traffic through the kernel (hence the OP's question). In Casey's answer above, I'm not sure if it was really necessary to have an external router or not above, but the following is completely self-contained. The two interfaces are eth2 and eth3.

Give IPs to the interfaces, and put them on separate networks:

ifconfig eth2 10.50.0.1/24
ifconfig eth3 10.50.1.1/24

Next we'll set up a double NAT scenario: two new fake networks used to reach the other. On the way out, source NAT to your fake network. On the way in, fix the destination. And vice versa for the other network:

# nat source IP 10.50.0.1 -> 10.60.0.1 when going to 10.60.1.1
iptables -t nat -A POSTROUTING -s 10.50.0.1 -d 10.60.1.1 -j SNAT --to-source 10.60.0.1

# nat inbound 10.60.0.1 -> 10.50.0.1
iptables -t nat -A PREROUTING -d 10.60.0.1 -j DNAT --to-destination 10.50.0.1

# nat source IP 10.50.1.1 -> 10.60.1.1 when going to 10.60.0.1
iptables -t nat -A POSTROUTING -s 10.50.1.1 -d 10.60.0.1 -j SNAT --to-source 10.60.1.1

# nat inbound 10.60.1.1 -> 10.50.1.1
iptables -t nat -A PREROUTING -d 10.60.1.1 -j DNAT --to-destination 10.50.1.1

Now tell the system how to get to each fake network, and prepopulate the arp entries (be sure to substitute your MAC addresses, don't use mine):

ip route add 10.60.1.1 dev eth2
arp -i eth2 -s 10.60.1.1 00:1B:21:C1:F6:0F # eth3's mac address

ip route add 10.60.0.1 dev eth3 
arp -i eth3 -s 10.60.0.1 00:1B:21:C1:F6:0E # eth2's mac address

This fools Linux enough to actually put packets onto the wire. For example:

ping 10.60.1.1

goes out eth2, the source IP 10.50.0.1 gets NATted to 10.60.0.1, and as it comes into eth3 the destination 10.60.1.1 gets NATted to 10.50.1.1. And the reply takes a similar journey.

Now use iperf to test throughput. Bind to the correct IPs, and be certain which IP you're contacting (the other end's fake address):

# server
./iperf -B 10.50.1.1 -s

# client: your destination is the other end's fake address
./iperf -B 10.50.0.1 -c 10.60.1.1 -t 60 -i 10

Make sure traffic is really going out to the wire:

tcpdump -nn -i eth2 -c 500

You can also watch /proc/interrupts just to be absolutely sure the card is being used:

while true ; do egrep 'eth2|eth3' /proc/interrupts ; sleep 1 ; done

Anyhow, I found this post searching for how to do this, thanks for the Q&A guys, and hope this helps anyone else finding this post in the future.