Hooking Linux machine to secondary router/ISP: how to setup routing correctly?
Abstract:
I have (I presume routing) issues when adding a secondary ISP to my existing network setup: incoming traffic through Router1 is not answered, but local traffic and incoming through Router0 is working fine.
How can I keep the parts that currently function well working, while making incoming traffic through Router1 work?
Elaboration:
I've sketched a diagram below with the bare essentials of the situation (in practice there are more devices on each LAN, but they do not matter).
This is the situation:
- I have two internal networks:
LAN0is192.168.x.0/24andLAN1is192.168.y.0/24. Both work fine for internal traffic (for instance http using cURL). -
LAN0has always been connected throughRouter0andISP0to theInternet. -
LAN1always hadRouter1, but is now connected throughISP1to theInternet. - Machines only on
LAN0and having a default route throughRouter0work fine for outgoing and incoming traffic. - Machines only on
LAN1and having a default route throughRouter1work fine for outgoing and incoming traffic. - Internal traffic on
LAN0andLAN1has always worked fine. - Incoming traffic through
Router1forWindowsBarrives correctly: I can connect to it over RDP fromWindowsC. - Incoming traffic through
Router1forLinuxBarrives (according to tcpdump), but not answered back as acurl http://e.f.g.hfronLinuxCshows with a tcpdump onLinuxBshows:
It shows only packets that - according to the tcpdump output format - have a SYN flag set:
LinuxB:/tmp/LinuxB.eth1.80 # tcpdump -i eth1 'port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
13:35:19.489779 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047182 ecr 0,sackOK,eol], length 0
13:35:19.788841 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047478 ecr 0,sackOK,eol], length 0
13:35:19.888835 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047578 ecr 0,sackOK,eol], length 0
13:35:19.989412 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047678 ecr 0,sackOK,eol], length 0
13:35:20.089685 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047778 ecr 0,sackOK,eol], length 0
13:35:20.190836 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047877 ecr 0,sackOK,eol], length 0
13:35:20.392123 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287048072 ecr 0,sackOK,eol], length 0
13:35:20.693692 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:21.197162 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:22.204134 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:24.115961 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:27.852374 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:31.967049 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
This is the LinuxB route table:
LinuxB:/tmp/LinuxB.eth1.80 # route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.x.1 0.0.0.0 UG 0 0 0 eth0
loopback * 255.0.0.0 U 0 0 0 lo
link-local * 255.255.0.0 U 0 0 0 eth0
192.168.x.0 * 255.255.255.0 U 0 0 0 eth0
192.168.x.0 * 255.255.255.0 U 0 0 0 eth1
Since connecting over RDP from WindowsC to WindowsB works fine, I resume this is indeed a routing problem. This is the WindowsB route table:
C:\temp>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 0c 29 35 77 e1 ...... AMD PCNET Family PCI Ethernet Adapter - Packet Scheduler Miniport
0x3 ...00 0c 29 35 77 eb ...... VMware Accelerated AMD PCNet Adapter - Packet Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.x.1 192.168.x.4 10
0.0.0.0 0.0.0.0 192.168.y.1 192.168.y.4 5
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
192.168.x.0 255.255.255.0 192.168.x.4 192.168.x.4 10
192.168.x.4 255.255.255.255 127.0.0.1 127.0.0.1 10
192.168.x.255 255.255.255.255 192.168.x.4 192.168.x.4 10
192.168.y.0 255.255.255.0 192.168.y.4 192.168.y.4 10
192.168.y.4 255.255.255.255 127.0.0.1 127.0.0.1 10
192.168.y.255 255.255.255.255 192.168.y.4 192.168.y.4 10
224.0.0.0 240.0.0.0 192.168.x.4 192.168.x.4 10
224.0.0.0 240.0.0.0 192.168.y.4 192.168.y.4 10
255.255.255.255 255.255.255.255 192.168.x.4 192.168.x.4 1
255.255.255.255 255.255.255.255 192.168.y.4 192.168.y.4 1
Default Gateway: 192.168.y.1
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.y.1 5
0.0.0.0 0.0.0.0 192.168.x.1 10
So how can I get the routing on LinuxB to be like this:
- keep default route on
LinuxBto192.168.x.1so outgoing traffic keeps usingRouter0/ISP0 - keep answering incoming requests coming from
LAN0onLAN0 - keep answering incoming requests coming from
LAN1onLAN1 - keep answering incoming requests through
Router0(a.b.c.d/192.168.x.1) via192.168.x.1 - start answering incoming requests through
Router1(e.f.g.h/192.168.y.1) via192.168.y.1 - bonus: have
Router1fail-over or load-balance withRouter0
Postscript:
The PNG image below is generated on UML text through the free online PlantUML engine. If you want to see the original UML text, paste the PNG image link into this PlantUML form, then press Submit.
I had a shell script for doing something like that a long-long time ago but, sorry, could find it. So I can only give you the pointers to the solutions I implemented back then. I'm writting mostly from memory so some examples are missing:
-
I had one routing table per uplink (ip route ... table 101, ip route ... table 102). That goes into /etc/iproute2/rt_tables.
101 isp1 102 isp2
You need to setup those tables also:
ip route add default via $Gateway1 dev $Interface1 table isp1 ip route add default via $Gateway2 dev $Interface2 table isp2
#Don't forget the default table:
ip route add default via $DefaultGateway dev $DefaultInterface
Enable iptables connection tracking (modprobe nf_conntrack)
- Set an iptables rule for NEW incoming connections to -j MARK the packets somehow (ie: 0x201, 0x202)
-
Set an ip rule that ensure that traffic going out trhough an interface uses the right routing table
ip rule add from $Ip1 table isp1 ip rule add from $Ip2 table isp2
Set an ip rule (ip rule add ...) stating that "packets marked 0x201 shall lookup routing table 201", one rule for every uplink.
With everything in place you should be able to receive and initiate connections with any wan uplink and even balance outgoing connections.
Those would be the basics. Iptables + "ip route" + "ip rule" and you're good to go.