Linux router: ping doesn't route back
I have a Debian box which I'm trying to set up as a router and an Ubuntu box which I'm using as a client.
My problem is that when the Ubuntu client tries to ping a server on the Internet, all the packets are lost (though, as you can see below, they seem to go to the server and back without problem).
I'm doing this in the Ubuntu Box:
# ping -I eth1 my.remote-server.com
PING my.remote-server.com (X.X.X.X) from 10.1.1.12 eth1: 56(84) bytes of data.
^C
--- my.remote-server.com ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12094ms
(I changed the name and IP of the remote server for privacy).
From the Debian Router I see this:
# tcpdump -i eth1 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 7, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 8, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 8, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 9, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 9, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 10, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 10, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 11, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 11, length 64
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel
# tcpdump -i eth2 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 213, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 213, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 214, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 214, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 215, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 215, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 216, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 216, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 217, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 217, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
And at the remote server I see this:
# tcpdump -i eth0 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 1, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 1, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 2, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 2, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 3, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 3, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 4, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 4, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 5, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 5, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 6, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 6, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 7, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 7, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 8, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 8, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 9, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 9, length 64
18 packets captured
228 packets received by filter
92 packets dropped by kernel
Here "X.X.X.X" is my remote server's IP and "Y.Y.Y.Y" is my local network's public IP. So, what I understand is that the ping packets are coming out of the Ubuntu box (10.1.1.12), to the router (10.1.1.1), from there to the next router (192.168.1.1) and reaching the remote server (X.X.X.X). Then they come back all the way to the Debian router, but they never reach the Ubuntu box back.
What am I missing?
Here's the Debian router setup:
# ifconfig
eth1 Link encap:Ethernet HWaddr 94:0c:6d:82:0d:98
inet addr:10.1.1.1 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::960c:6dff:fe82:d98/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:105761 errors:0 dropped:0 overruns:0 frame:0
TX packets:48944 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:40298768 (38.4 MiB) TX bytes:44831595 (42.7 MiB)
Interrupt:19 Base address:0x6000
eth2 Link encap:Ethernet HWaddr 6c:f0:49:a4:47:38
inet addr:192.168.1.10 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::6ef0:49ff:fea4:4738/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:38335992 errors:0 dropped:0 overruns:0 frame:0
TX packets:37097705 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:1000
RX bytes:4260680226 (3.9 GiB) TX bytes:3759806551 (3.5 GiB)
Interrupt:27
eth3 Link encap:Ethernet HWaddr 94:0c:6d:82:c8:72
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:20 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3408 errors:0 dropped:0 overruns:0 frame:0
TX packets:3408 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:358445 (350.0 KiB) TX bytes:358445 (350.0 KiB)
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2767779 errors:0 dropped:0 overruns:0 frame:0
TX packets:1569477 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:3609469393 (3.3 GiB) TX bytes:96113978 (91.6 MiB)
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.8.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
127.0.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 lo
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
192.168.1.0 0.0.0.0 255.255.255.0 U 1 0 0 eth2
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth2
# arp -n
# Note: Here I have changed all the different MACs except the ones corresponding to the Ubuntu box (on 10.1.1.12 and 192.168.1.12)
Address HWtype HWaddress Flags Mask Iface
192.168.1.118 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.72 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.94 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.102 ether NN:NN:NN:NN:NN:NN C eth2
10.1.1.12 ether 00:1e:67:15:2b:f0 C eth1
192.168.1.86 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.2 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.61 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.64 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.116 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.91 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.52 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.93 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.87 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.92 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.100 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.40 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.53 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.1 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.83 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.89 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.12 ether 00:1e:67:15:2b:f1 C eth2
192.168.1.77 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.66 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.90 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.65 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.41 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.78 ether NN:NN:NN:NN:NN:NN C eth2
192.168.1.123 ether NN:NN:NN:NN:NN:NN C eth2
# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 10.1.1.0/24 !10.1.1.0/24
MASQUERADE all -- !10.1.1.0/24 10.1.1.0/24
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
And here's the Ubuntu box:
# ifconfig
eth0 Link encap:Ethernet HWaddr 00:1e:67:15:2b:f1
inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::21e:67ff:fe15:2bf1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:28785139 errors:0 dropped:0 overruns:0 frame:0
TX packets:19050735 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:32068182803 (32.0 GB) TX bytes:6061333280 (6.0 GB)
Interrupt:16 Memory:b1a00000-b1a20000
eth1 Link encap:Ethernet HWaddr 00:1e:67:15:2b:f0
inet addr:10.1.1.12 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::21e:67ff:fe15:2bf0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:285086 errors:0 dropped:0 overruns:0 frame:0
TX packets:12719 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:30817249 (30.8 MB) TX bytes:2153228 (2.1 MB)
Interrupt:16 Memory:b1900000-b1920000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:86048 errors:0 dropped:0 overruns:0 frame:0
TX packets:86048 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11426538 (11.4 MB) TX bytes:11426538 (11.4 MB)
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.1.1.1 0.0.0.0 UG 100 0 0 eth1
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.8.0.0 192.168.1.10 255.255.255.0 UG 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
# arp -n
# Note: Here I have changed all the different MACs except the ones corresponding to the Debian box (on 10.1.1.1 and 192.168.1.10)
Address HWtype HWaddress Flags Mask Iface
192.168.1.70 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.90 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.97 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.103 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.13 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.120 (incomplete) eth0
192.168.1.111 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.118 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.51 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.102 (incomplete) eth0
192.168.1.64 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.52 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.74 (incomplete) eth0
192.168.1.94 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.121 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.72 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.87 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.91 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.71 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.78 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.83 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.88 (incomplete) eth0
192.168.1.82 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.98 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.100 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.93 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.73 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.11 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.85 (incomplete) eth0
192.168.1.112 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.89 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.65 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.81 ether NN:NN:NN:NN:NN:NN C eth0
10.1.1.1 ether 94:0c:6d:82:0d:98 C eth1
192.168.1.53 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.116 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.61 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.10 ether 6c:f0:49:a4:47:38 C eth0
192.168.1.86 (incomplete) eth0
192.168.1.119 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.66 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.1 ether NN:NN:NN:NN:NN:NN C eth0
192.168.1.1 ether NN:NN:NN:NN:NN:NN C eth1
192.168.1.92 ether NN:NN:NN:NN:NN:NN C eth0
# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Edit: Following Patrick's suggestion, I did a tcpdump con the Ubuntu box and I see this:
# tcpdump -i eth1 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 1, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 1, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 2, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 2, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 3, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 3, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 4, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 4, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 5, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 5, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 6, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 6, length 64
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
So the question is: if all packets seem to be coming and going, why does ping report 100% packet loss?
From your question in the comments:
On the remote server I see requests and replies. But on the Debian router I don't see anything... on none of the interfaces! My guess is that now, the Ubuntu box is talking directly to the router on 192.168.1.1 THOUGH sending requests with IP 10.1.1.12, so it can't route back. But why??
From the Ubuntu server:
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 <---
0.0.0.0 10.1.1.1 0.0.0.0 UG 100 0 0 eth1
At the time you captured this routing table, you have a lower metric default through eth0 pointing to your router at 192.168.1.1 (i.e. not the debian machine). A lower metric default is always followed first, which means Ubuntu wants to send all non-connected traffic directly to 192.168.1.1.
When you have downtime available, please remove that default with
route del default gw 192.168.1.1 dev eth0
I'm still simmering on the bigger problem (original sniffer traces show ping replies on Ubuntu:eth1, but no pings accepted by the OS). Could you please ping from Ubuntu:eth1 and simultaneously capture on Debian:eth2 to demonstrate what is happening with NAT after you force the Ubuntu to send all traffic through the Debian again?
Did you check if reverse path filtering is enabled on the Ubuntu box?
It's a sysctl setting (net.ipv4.conf.all.rp_filter), it will filter incoming packets if the source address is coming in on the "wrong" interface ( i.e. not the interface that the kernel would route it to )
You could also try net.ipv4.conf.all.log_martians=1 to try to see what's happening.
The key to make this work is to create separate routing tables for the different interfaces, and tell the networking stack to use these routing tables instead of the default one.
In your case this should make ping -I eth2 8.8.8.8 work:
# register the 'foo' table name and give it id 1
echo '1 foo' >> /etc/iproute2/rt_tables
# setup routing table 'foo'
ip route add 192.168.1.0/24 dev eth2 src 192.168.1.10 table foo
ip route add default via 192.168.1.1 table foo
# use routing table 'foo' for address 192.168.1.10
ip rule add from 192.168.1.10 table foo
More information on routing for multiple uplinks can be found in the LARTC HOWTO: http://lartc.org/howto/lartc.rpdb.multiple-links.html