Route return traffic to correct gateway depending on service

On my office network I have two internet connections and one CentOS server running a website (HTTPS on port 443). The website should be publicly accessible through the public IP of the first internet connection (ISP-1). The other internet connection, ISP-2, id the default gateway on the network. Both internet connections have routers (the household-kind) with NAT, SPI firewalls etc. The router on ISP-2 is a Netgear WNDR3700 (aka N600) with original firmware.

The problem is that the website is unreachable. Looks like incoming traffic on ISP-1 will reach the server but the returning traffic is routed through ISP-2, effectively making the site unreachable. As far as I can tell I can't do port based routing on the WNDR3700.

What are my options to make this work? I've been looking at implementing an iptables / routing based solution on the server itself but haven't been able to make that work.

Update: Note that the server has one network interface connecting it to both routers.

If I am understanding your intentions correctly, you want your webserver to normally use ISP-2 as its default gateway for outgoing traffic, with the exception of its responses to external web requests, which must transit via ISP-1 instead. Here is a sketch of a solution using policy routing:

echo "101 webtraffic" >> /etc/iproute2/rt_tables

ip route add default table webtraffic via $ISP1_GW_LAN_IP
ip rule add fwmark 1 table webtraffic
iptables -t mangle -A OUTPUT -d \! $LAN_NET_PREFIX \
                             -p tcp -m tcp --sport 443 \
                             -j MARK --set-mark 1

where:

• LAN_NET_PREFIX is your LAN's network prefix (e.g. 192.168.100.0/24), and
• ISP1_GW_LAN_IP is the LAN IP address of your gateway to ISP-1 (e.g. 192.168.100.100).

The first ip command sets the default route on the webtraffic table to your ISP-1 gateway, and the second ensures that packets marked 1 are routed using the webtraffic table. Finally, the iptables rule marks the appropriate outgoing packets, ensuring that their next hop will be towards ISP-1.

Here is an alternate solution that uses an experimental iptables module, the ROUTE target:

iptables -t mangle -A POSTROUTING -d \! $LAN_NET_PREFIX \
                                  -p tcp -m tcp --sport 443 \
                                  -j ROUTE --gw $ISP1_GW_LAN_IP

This rule would override the routing decision for outgoing web response packets, sending them to your ISP-1 gateway, instead of to the default ISP-2. All other traffic, including web responses to clients on your LAN, would not be affected. As has been pointed out in the comments, the ROUTE target is very likely not to be implemented on any system that has not explicitly patched it into the kernel, since it is experimental.

I had the same issue, but I solved only with iproute2 (source routing). Marking with iptables wasn't necessary:

echo "101 webtraffic" >> /etc/iproute2/rt_tables
ip route add default table webtraffic via $ISP1_GW_LAN_IP
ip rule add from $ISP1_IP table webtraffic

The point is that not only web may use the ISP1 connection. You may choose. It's good because you may connect to the server from ssh from both connections if someone fails. As a CentOS user I created the following files so my changes weren't lost after reboot:

echo "default table webtraffic via $ISP1_GW_LAN_IP" >> /etc/sysconfig/network-scripts/route-eth1
echo "from $ISP1_IP table webtraffic" >> /etc/sysconfig/network-scripts/rule-eth1