Linux router: ping doesn't route back

I have a Debian box which I'm trying to set up as a router and an Ubuntu box which I'm using as a client.

My problem is that when the Ubuntu client tries to ping a server on the Internet, all the packets are lost (though, as you can see below, they seem to go to the server and back without problem).

I'm doing this in the Ubuntu Box:

# ping -I eth1 my.remote-server.com
PING my.remote-server.com (X.X.X.X) from 10.1.1.12 eth1: 56(84) bytes of data.
^C
--- my.remote-server.com ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12094ms

(I changed the name and IP of the remote server for privacy).

From the Debian Router I see this:

# tcpdump -i eth1 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 7, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 8, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 8, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 9, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 9, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 10, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 10, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 305, seq 11, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 305, seq 11, length 64
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel

# tcpdump -i eth2 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 213, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 213, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 214, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 214, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 215, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 215, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 216, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 216, length 64
IP 192.168.1.10 > X.X.X.X: ICMP echo request, id 360, seq 217, length 64
IP X.X.X.X > 192.168.1.10: ICMP echo reply, id 360, seq 217, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

And at the remote server I see this:

# tcpdump -i eth0 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 1, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 1, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 2, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 2, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 3, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 3, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 4, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 4, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 5, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 5, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 6, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 6, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 7, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 7, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 8, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 8, length 64
IP Y.Y.Y.Y > X.X.X.X: ICMP echo request, id 360, seq 9, length 64
IP X.X.X.X > Y.Y.Y.Y: ICMP echo reply, id 360, seq 9, length 64

18 packets captured
228 packets received by filter
92 packets dropped by kernel

Here "X.X.X.X" is my remote server's IP and "Y.Y.Y.Y" is my local network's public IP. So, what I understand is that the ping packets are coming out of the Ubuntu box (10.1.1.12), to the router (10.1.1.1), from there to the next router (192.168.1.1) and reaching the remote server (X.X.X.X). Then they come back all the way to the Debian router, but they never reach the Ubuntu box back.

What am I missing?

Here's the Debian router setup:

# ifconfig
eth1      Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.1.1  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::960c:6dff:fe82:d98/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:105761 errors:0 dropped:0 overruns:0 frame:0
          TX packets:48944 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:40298768 (38.4 MiB)  TX bytes:44831595 (42.7 MiB)
          Interrupt:19 Base address:0x6000 

eth2      Link encap:Ethernet  HWaddr 6c:f0:49:a4:47:38  
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::6ef0:49ff:fea4:4738/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38335992 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37097705 errors:0 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:1000 
          RX bytes:4260680226 (3.9 GiB)  TX bytes:3759806551 (3.5 GiB)
          Interrupt:27 

eth3      Link encap:Ethernet  HWaddr 94:0c:6d:82:c8:72  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20 Base address:0x2000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3408 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:358445 (350.0 KiB)  TX bytes:358445 (350.0 KiB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:2767779 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1569477 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:3609469393 (3.3 GiB)  TX bytes:96113978 (91.6 MiB)


# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
127.0.0.1       0.0.0.0         255.255.255.255 UH    0      0        0 lo
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
192.168.1.0     0.0.0.0         255.255.255.0   U     1      0        0 eth2
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth2

# arp -n 
# Note: Here I have changed all the different MACs except the ones corresponding to the Ubuntu box (on 10.1.1.12 and 192.168.1.12)
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.118            ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.72             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.94             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.102            ether   NN:NN:NN:NN:NN:NN   C                     eth2
10.1.1.12                ether   00:1e:67:15:2b:f0   C                     eth1
192.168.1.86             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.2              ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.61             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.64             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.116            ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.91             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.52             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.93             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.87             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.92             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.100            ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.40             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.53             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.1              ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.83             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.89             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.12             ether   00:1e:67:15:2b:f1   C                     eth2
192.168.1.77             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.66             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.90             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.65             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.41             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.78             ether   NN:NN:NN:NN:NN:NN   C                     eth2
192.168.1.123            ether   NN:NN:NN:NN:NN:NN   C                     eth2


# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination   

# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  10.1.1.0/24         !10.1.1.0/24         
MASQUERADE  all  -- !10.1.1.0/24          10.1.1.0/24         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

And here's the Ubuntu box:

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1e:67:15:2b:f1  
          inet addr:192.168.1.12  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:67ff:fe15:2bf1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28785139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19050735 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:32068182803 (32.0 GB)  TX bytes:6061333280 (6.0 GB)
          Interrupt:16 Memory:b1a00000-b1a20000 

eth1      Link encap:Ethernet  HWaddr 00:1e:67:15:2b:f0  
          inet addr:10.1.1.12  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:67ff:fe15:2bf0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:285086 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12719 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:30817249 (30.8 MB)  TX bytes:2153228 (2.1 MB)
          Interrupt:16 Memory:b1900000-b1920000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:86048 errors:0 dropped:0 overruns:0 frame:0
          TX packets:86048 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:11426538 (11.4 MB)  TX bytes:11426538 (11.4 MB)

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
0.0.0.0         10.1.1.1        0.0.0.0         UG    100    0        0 eth1
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.8.0.0        192.168.1.10    255.255.255.0   UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0

# arp -n
# Note: Here I have changed all the different MACs except the ones corresponding to the Debian box (on 10.1.1.1 and 192.168.1.10)
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.70             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.90             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.97             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.103            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.13             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.120                    (incomplete)                              eth0
192.168.1.111            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.118            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.51             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.102                    (incomplete)                              eth0
192.168.1.64             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.52             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.74                     (incomplete)                              eth0
192.168.1.94             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.121            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.72             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.87             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.91             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.71             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.78             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.83             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.88                     (incomplete)                              eth0
192.168.1.82             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.98             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.100            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.93             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.73             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.11             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.85                     (incomplete)                              eth0
192.168.1.112            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.89             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.65             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.81             ether   NN:NN:NN:NN:NN:NN   C                     eth0
10.1.1.1                 ether   94:0c:6d:82:0d:98   C                     eth1
192.168.1.53             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.116            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.61             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.10             ether   6c:f0:49:a4:47:38   C                     eth0
192.168.1.86                     (incomplete)                              eth0
192.168.1.119            ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.66             ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.1              ether   NN:NN:NN:NN:NN:NN   C                     eth0
192.168.1.1              ether   NN:NN:NN:NN:NN:NN   C                     eth1
192.168.1.92             ether   NN:NN:NN:NN:NN:NN   C                     eth0

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination  

# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination 

Edit: Following Patrick's suggestion, I did a tcpdump con the Ubuntu box and I see this:

# tcpdump -i eth1 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 1, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 1, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 2, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 2, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 3, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 3, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 4, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 4, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 5, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 5, length 64
IP 10.1.1.12 > X.X.X.X: ICMP echo request, id 21967, seq 6, length 64
IP X.X.X.X > 10.1.1.12: ICMP echo reply, id 21967, seq 6, length 64
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

So the question is: if all packets seem to be coming and going, why does ping report 100% packet loss?


From your question in the comments:

On the remote server I see requests and replies. But on the Debian router I don't see anything... on none of the interfaces! My guess is that now, the Ubuntu box is talking directly to the router on 192.168.1.1 THOUGH sending requests with IP 10.1.1.12, so it can't route back. But why??

From the Ubuntu server:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0 <---
0.0.0.0         10.1.1.1        0.0.0.0         UG    100    0        0 eth1

At the time you captured this routing table, you have a lower metric default through eth0 pointing to your router at 192.168.1.1 (i.e. not the debian machine). A lower metric default is always followed first, which means Ubuntu wants to send all non-connected traffic directly to 192.168.1.1.

When you have downtime available, please remove that default with

route del default gw 192.168.1.1 dev eth0

I'm still simmering on the bigger problem (original sniffer traces show ping replies on Ubuntu:eth1, but no pings accepted by the OS). Could you please ping from Ubuntu:eth1 and simultaneously capture on Debian:eth2 to demonstrate what is happening with NAT after you force the Ubuntu to send all traffic through the Debian again?


Did you check if reverse path filtering is enabled on the Ubuntu box?

It's a sysctl setting (net.ipv4.conf.all.rp_filter), it will filter incoming packets if the source address is coming in on the "wrong" interface ( i.e. not the interface that the kernel would route it to )

You could also try net.ipv4.conf.all.log_martians=1 to try to see what's happening.


The key to make this work is to create separate routing tables for the different interfaces, and tell the networking stack to use these routing tables instead of the default one.

In your case this should make ping -I eth2 8.8.8.8 work:

# register the 'foo' table name and give it id 1
echo '1 foo' >> /etc/iproute2/rt_tables

# setup routing table 'foo'
ip route add 192.168.1.0/24 dev eth2 src 192.168.1.10 table foo
ip route add default via 192.168.1.1 table foo

# use routing table 'foo' for address 192.168.1.10
ip rule add from 192.168.1.10 table foo

More information on routing for multiple uplinks can be found in the LARTC HOWTO: http://lartc.org/howto/lartc.rpdb.multiple-links.html