Packets returning but traceroute fails

Solution 1:

Wireshark will show what arrives on the network interface. The kernel has obviously seen those packets, but for some reason decided that they are not to be delivered to the traceroute command.

There is a few things that could have gone wrong causing the kernel to decide not to deliver those packets.

  • You may have an asymmetrical routing which is not suitable for reverse path filtering, but have left rp_filter enabled.
  • The kernel may not be able to match the content of the ICMP error message with a local socket. This could happen due to the packet having been truncated with insufficient information available to make such decision. This could also happen due to some broken NAT configuration where packets in one direction gets routed through a NAT but not in the other direction.
  • The kernel may drop the packets due to bad checksum.

Of those I think the rp_filter sounds like the most likely explanation. You didn't specify an operating system, but it looks like it might be a Linux system, so try this command: head /proc/sys/net/ipv4/conf/*/rp_filter. You would likely see 1 on every one of them, meaning the filter is enabled. Try writing a 0 to the one corresponding to the interface the packets are being dropped from as well as to the all device name.