linux drops packet when trying to route it - why?

I track packets using LOG targets for all chains of all tables and the last chain I see my packet in is the PREROUTING chain of the mangle table:

mangle_PREROUTING: IN=eth0 OUT= MAC=00:16:3e:0f:01:00:00:16:3e:0f:02:00:08:00
SRC=10.2.1.2 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=60912 PROTO=ICMP
TYPE=0 CODE=0 ID=41230 SEQ=1

Per http://inai.de/images/nf-packet-flow.png, I expected that packet to appear in FORWARD chain of the mangle table, but it never gets there. The only thing between these is the routing table.

# ip rule list
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# ip route list
default via 192.168.178.100 dev eth3
10.1.1.0/24 dev eth1  proto kernel  scope link  src 10.1.1.1
10.1.2.0/24 dev eth2  proto kernel  scope link  src 10.1.2.1
10.5.0.0/24 dev eth0  proto kernel  scope link  src 10.5.0.1
192.168.178.0/24 dev eth3  proto kernel  scope link  src 192.168.178.1

So that packet should be routable via 10.1.1.1. (That system can also ping 10.1.1.2.)

iptables (all chains ACCEPT) has only these rules (except for logging at the end):

-t raw -A PREROUTING -j MARK --set-xmark 0x0/0xffffffff
-t mangle -A PREROUTING -s 10.1.1.0/24 -d 10.2.1.0/24 -j MARK --set-xmark 0x1/0xffffffff
-t mangle -A PREROUTING -p esp -j MARK --set-xmark 0x1/0xffffffff

But those rules should not match: The destination is 10.1.1.2 and it is not ESP...

Any hints?

Best regards, Steffen

I think that your problem is the rp_filter. You have a packet with source 10.2.1.2 that was received in the interface eth0. Your route table say the network 10.2.1.2 is reached via eth3 (default route). I suppose that you don't have a typo.

If the problem is the rp_filter, packets are dropped before FORWARD.

rp_filter means "reverse path filter", usually enabled by default. Only the packets incomming from the right interface (acordding to the routes) are allowed.