Forwarded packets are visible by tcpdump but not received by application
I have 3 machines:
- A 10.8.0.8/24
- B 10.8.0.13/24 10.248.0.34/23
- C 10.8.0.40/24 10.248.0.12/23
On A I add a route from A to C using: ip route add 10.248.0.12 via 10.8.0.13.
Then I test the configuration. On C I type nc -ul 2002. Then on A I use nc -u 10.248.0.12 2002 to send some packets.
Packets are not received by nc on C.
However when I use on Ctcpdump -n "(src host 10.8.0.8 and dst host 10.248.0.12) or (src host 10.248.0.12 and dst host 10.8.0.8)" -vv -i any
then it reads15:19:46.756649 IP (tos 0x0, ttl 63, id 47699, offset 0, flags [DF], proto UDP (17), length 29) 10.8.0.8.44254 > 10.248.0.12.2002: [udp sum ok] UDP, length 1.
What can I do to inspect what happens with the packets, clearly the packets are routed from A to C, but for some reason they are rejected.
On C:
iptables -t filter -L FORWARD -n
Chain FORWARD (policy ACCEPT)
target prot opt source destination
WEAVE-NPC all -- 0.0.0.0/0 0.0.0.0/0 /* NOTE: this must go before '-j KUBE-FORWARD' */
NFLOG all -- 0.0.0.0/0 0.0.0.0/0 state NEW nflog-group 86
DROP all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* generated for LXD network fanbr0 */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* generated for LXD network fanbr0 */
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
CHECKSUM udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68 /* generated for LXD network fanbr0 */ CHECKSUM fill
As pointed out by @A.B the explanation is reverse path filtering. I made 10.248.0.12 accessible from A by disabling rp filtering on C.
sysctl -w net.ipv4.conf.all.rp_filter=0echo 0 > /proc/sys/net/ipv4/conf/*/rp_filter