routing traffic between two network cards through firewall

Solution 1:

Use the network namespace solution. Definitely. It's asking for it.

Trying to do anything else is bound to fail, unless you have the knowledge to mess around with the local table and policy routing to make it work. Sending arbitrary crap on an interface is one thing, having the other interface respond the way you want is the real problem here: the kernel will immediately recognize its own IP in the local routing table, which is the first table to be consulted by default (see ip rule). Note that the man pages say (or used to say) that this table cannot be modified and the rule cannot be changed, but it can be done, but you mileage may vary.

So to make it work without namespaces, you need these ugly hacks, which are not guaranteed to work (and which i have not tested):

1. Move the local table to a higher priority. Note that the man pages says this is not possible.

   ip rule add prio 100 lookup local
   ip rule del prio 0
   

   Note that messing around with this rule is a quick way to break all network connectivity. If the local table isn't present, the kernel won't recognize any of its IP address, so you will not even ping 127.0.0.1.

2. Create two routing tables for your two interfaces. let's decide that table 10 is for eth0 and 11 eth1. Create a route to the other end.

   ip route add eth1's ip dev eth0 table 10
   ip route add eth0's ip dev eth1 table 11
   

3. Now add the rules for these tables. Anything that we send (iif lo, as an exception, means 'generated locally') from eth0's IP should use table 10. Make sure that the priority is less than the lookup local's priority.

   ip rule add from eth0's_ip iif lo lookup 10 prio 10
   ip rule add from eth1's_ip iif lo lookup 11 prio 11
   

This should be enough to do what you want to do, at least on an IP level.

Just to understand what happens:

• When the kernel receives a icmp echo packet from the eth0 interface from eth1's IP to eth0's IP, it will first lookup the destination address to decide if this destination IP is its own (or, if not, where the packet should be forwarded). So it looks into our two rules, see that they don't match (the packets are not being sent, they are being received, so iif lo fails), then fallback on the local table, which is the one which used to be first in the first place. That local table tells us that eth0's IP actually belong to this host, so it can process it.
• The kernel generates a response, then proceed to route this response. The response is an icmp echo reply packet that is generated locally, with eth0's IP as the source address and eth1's IP as the destination address. So our rule from eth0's_IP iif lo matches, the table 10 is consulted, which says to route eth1's IP via eth0. Had our rule not existed, the kernel would have looked up the local table, which says that eth1's IP belongs to this host, so the packet would have gone through the loopback interface instead.

Reality is a bit more complicated than that. For example, the kernel is often configured to do reverse path filtering, so it would check, upon receiving the packet from eth0, that if we invert the source and destination address and perform routing, that the resulting packet should be routed via eth0, and if not, then the address are deemed forged and the packet is discarded. This is not a problem if your routes are symmetric, which, fortunately in this case, are. The kernel could also consult the local table directly without consulting the policy routing database (ip rule) as an optimization. If it does that at any point while processing the packet, then you are screwed.

EDIT: if you are really trying to do this, you might want to activate the sysctl knob "accept_local". This is one case where the kernel looks up the local table directly without asking.

So really, don't do this. Use the network namespace solution. It's guaranteed to work. Just move one of the interfaces to the other namespace, and configure each namespace like they were two separate hosts talking through that firewall.