Load balancing & NAT-ing multiple ISP connections on Linux

I have done load balancing using both lartc.org and iptables methods, and I find that the iptables method is easier to understand and implement. The only downside is that you need a fairly recent iptables version to be able to use statistic module

Let's suppose a few things:

LAN: eth0: 192.168.0.1/24

ISP1: eth1: 192.168.1.1/24, gateway: 192.168.1.2/24

ISP2: eth2: 192.168.2.1/24, gateway: 192.168.2.2/24

So here is how I would do by using iptables method:

Route tables

First edit the /etc/iproute2/rt_tables to add a map between route table numbers and ISP names

...
10 ISP1
20 ISP2
...

So table 10 and 20 is for ISP1 and ISP2, respectively. I need to populate these tables with routes from main table with this code snippet (which I have taken from hxxp://linux-ip.net/html/adv-multi-internet.html)

ip route show table main | grep -Ev '^default' \
   | while read ROUTE ; do
     ip route add table ISP1 $ROUTE
done

And add default gateway to ISP1 through that ISP1's gateway:

ip route add default via 192.168.1.2 table ISP1

Do the same for ISP2

So now I have 2 route tables, 1 for each ISP.

Iptables

OK now I use iptables to evenly distribute packets to each route tables. More info on how this work can be found here (http://www.diegolima.org/wordpress/?p=36) and here (http://home.regit.org/?page_id=7)

# iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
# iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
# iptables -t mangle -A PREROUTING -j MARK --set-mark 10
# iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
# iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

NAT

Well NAT is easy:

# iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

mefat's answer helped me a lot but rather than a one off copy of all the main table rules into the two ISP tables a better approach may be to use the rule prio to add the default rules after the main table.

Set up /etc/iproute2/rt_tables as normal:

...
10 ISP1
20 ISP2
...

Note that

ip rule show

Shows rules 0->local, 32766->main and 32767->default. See man ip for more details.

Crucially the routing process will work from low prio to high prio rules ... but 32767 is not the highest rule#. So if the main routing table has no default route (but may contain all kinds of dynamically changing routes for vpns etc) then if a match is not made, it falls through to default (normally empty) and then looks for higher prio rules.

See the 'throw' section here: http://linux-ip.net/html/routing-tables.html

So now setup

ip route add default dev $ISP1_IFACE table ISP1
ip route add default dev $ISP2_IFACE table ISP2

and to make sure they're looked at after the main table:

ip rule add fwmark 20 table ISP1 prio 33000
ip rule add fwmark 10 table ISP2 prio 33000

Use

ip rule show

again to verify that the these rules are higher prio than main

Then use CONNMARK mangling as mefat said:

# iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
# iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
# iptables -t mangle -A PREROUTING -j MARK --set-mark 10
# iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
# iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

Things to note: pppd needs nodefaultroute otherwise it sets up in main; when a device restarts the ISP1/ISP2 tables are cleaned so need to be restored using a script.

I use a script in /etc/ppp/ip-{up,down}.d/dual-routing

# One of my connections is ~2x faster than the other
BALANCED=0.3
ALL_ISP1=0
ALL_ISP2=1

RULENUM=4
set_balance() {
    iptables -t mangle -R PREROUTING $RULENUM -m statistic --mode random --probability $0 -j MARK --set-mark 2
}

# if both up
set_balance $BALANCED
# if ppp1 down:
set_balance $ALL_ISP1
# if ppp0 down:
set_balance $ALL_ISP2

This is connection based load-balancing so I'm going to look at using load to monitor and replace the stats rule: iptables -t mangle -R PREROUTING <n> from userspace. So if there's a long-running download on one connection and the other connection is lightly loaded we should prefer the lightly loaded connection.