iptables packet forwarding to one of two gateways depending on origin

Given the following installation:

old router  192.168.1.1     with NAT forward of tcp port 80 to 192.168.1.10:80
new router  192.168.1.2     with NAT forward of tcp port 80 to 192.168.1.10:80
web server  192.168.1.10    with default gateway 192.168.1.1

Currently, the DNS entry for my service points to the external address of the old router. The router does a port forwarding of the traffic to the web server which returns the answer back to the old gateway.

For a seamless migration to a new router (with another external IP), I want to setup a new router first, test the whole thing with both connections active, and then change the DNS IP to the new external address.

Now, with the above setup, the old router still works. But tcp connections which are addressed to the new router, are also answered to the old router which cannot handle them.

I thought about using nat with masquerading but then all traffic which is addressed to the new router will look like local traffic. This tricks out server ip-based filters and logging.

Now, my plan is to use a helper PC with Debian and iptables for a temporary solution:

old router  192.168.1.1     with NAT forward of tcp port 80 to 192.168.1.10:80
new router  192.168.1.2     with NAT forward of tcp port 80 to 192.168.1.20:80
web server  192.168.1.10    with default gateway 192.168.1.20
helper pc   192.168.1.20

the helper pc is now responsible for finding the correct gateway:

• keep state of connections which were forwarded by 192.168.1.2 and forward them without modifying to 192.168.1.10 (i think about identifying them by their origin mac address and SYN)
• forward packets of tracked connections to 192.168.1.2
• forward packets of not-tracked connections to 192.168.1.1

I found a lot of input in the internet and especially serverfault.com (especially this one), but all of them cover only a single part of this problem. I guess it needs a combination of mac address based filter rules -m mac --mac-source <mac>, NAT, states and rt_tables --set-mark / ip rule add fwmark (--gw doesn't seem to be supported by Debian).

There are two effective solutions, depending on the scenario.

Solution 1: The webserver is Linux and fully under your control.

First, ensure that /etc/iproute2/rt_tables contains the following lines:

201     gw1
202     gw2

Second, populate the tables

ip route add table gw1 default via $GW1_IP dev $ETH metric 100
ip route add table gw2 default via $GW2_IP dev $ETH metric 100

Third, create two iproute2 rules to 'shunt' processing to those custom tables:

ip rule add prio 100 from all fwmark 1 lookup gw1
ip rule add prio 110 from all fwmark 2 lookup gw2

Finally, create the set of iptables commands to mark the corresponding packets properly:

# Make sure mark exists before routing happens
#   The first handles incoming packets
-t mangle -A PREROUTING -j CONNMARK --restore-mark
#   The second handles outgoing packets
-t mangle -A OUTPUT     -j CONNMARK --restore-mark

# Mark packets and save the mark    
-t mangle -A INPUT -m mac --mac-source $GW1_MAC -j MARK --set-mark 1
-t mangle -A INPUT -m mac --mac-source $GW2_MAC -j MARK --set-mark 2
-t mangle -A INPUT -j CONNMARK --save-mark

Solution 2: The webserver is not Linux and/or it is not fully under your control

This solution is very similar to the previous solution. The difference is in the set of iptables rules:

# Make sure mark exists before routing happens
-t mangle -A PREROUTING -j CONNMARK --restore-mark

# Mark packets and save the mark    
-t mangle -A FORWARD -m mac --mac-source $GW1_MAC -j MARK --set-mark 1
-t mangle -A FORWARD -m mac --mac-source $GW2_MAC -j MARK --set-mark 2

-t mangle -A POSTROUTING -j CONNMARK --save-mark

Edit: Modified Solution 2

Still the same as the above, but add:

ip rule add prio 10 to 192.168.1.0/24 lookup main

this ensures that packets destined for the local network (I'm assuming 192.168.1.0/24) will not be handled by the gateways.

Also, add another iptables rule:

-t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.10

Finally, have both gateways forward port 80 to the 'helper pc'; it will save a great deal of headache trying to troubleshoot 'half-open' connections (because, in your current plan the 'helper pc' can only see traffic originating from the webserver instead of traffic going bothways).