How to achieve per-packet multipath routing on Linux?

Solution 1:

Ok, so after having had more time to investigate this I found a way to do it using Linux TEQL (True Link Equalizer). Here is a link I loosely followed, but with some tweaks.

http://lartc.org/howto/lartc.loadshare.html

This is how I got it working on Arch Linux ARMv7 (Raspberry Pi 3)

On boot:

The following command should be run on boot to load the appropriate Kernel module.

modprobe sch_teql

The following commands also to run on boot assuming you want to NAT from a local network on eth0.

sysctl -w net.ipv4.ip_forward=1
iptables -A INPUT -i ppp+ -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i ppp+ -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A POSTROUTING -t nat -o teql+ -j MASQUERADE

The FORWARD return traffic is on ppp+, and the POSTROUTING MASQUERADE on teql+ because the outgoing traffic goes out on teql and the return traffic comes back on ppp.

When ppp links come up:

Assuming the links to be load-balanced are ppp, the following commands to be run in a script in an /etc/ppp/ip-up.d/ script.

sysctl -w net.ipv4.conf.ppp1.rp_filter=2
sysctl -w net.ipv4.conf.ppp2.rp_filter=2
tc qdisc add dev ppp1 root teql0
tc qdisc add dev ppp2 root teql0
ip address add 1.1.1.1/32 dev teql0
# you can add additional public IP addresses teql0 if you need to
ip link set teql0 up
ip route replace default scope global dev teql0

Where 1.1.1.1 is your ISP-facing public IP address. Additional public IPs can be assigned to the teql0 device, but don't need to be assigned to the ppp devices. In my setup the two ppp links share the same IP (negotiated by pppoe etc.) The teql link it manually assigned as shown above. The ISP needs to send traffic for the IP equally down both links.

The reverse path (rp_filter) is set to 2 (loose) both in the script above so that the return packets are not dropped due to them coming back on the ppp interfaces rather than teql0.

I have set it up that way, and it works perfectly. Very easy! When the links fail, there is seamless failover. When they come up, they just start working again. Seems like there is no packet loss or delay when it fails over, and none when it comes back up either.

Also, one of the commenters suggested the below link which uses policy routing, with iptables to mark every other packet etc. but I will try in a few days to see whether it works any better than the above and provide feedback here accordingly.

http://support.aa.net.uk/Router_-_Linux_upload_bonding_using_policy_routing