Multiple NICs in linux: how to return route through established?
If there's a linux host with two interfaces on two different networks (the primary network, and the out-of-band management network, call them "primary" and "mgmt"), how can return traffic be routed back through the same interface from whence it came?
In other words: From where I sit, I go through a router to access either the primary or mgmt interfaces of the machine. When I ssh to the primary interface, no problem. When I ssh to the mgmt interface, my packet arrives at the server, but the server sends its response back through the default gateway which is on the primary interface. Hence, my connection is not established. How can I make the server respond to requests through the mgmt network, that were inbound received on the mgmt network?
Solution 1:
Source-based routing / ip rule.
Suppose primary has 1.2.3.4 and mgmt has 2.3.4.5 (and their default gateways are 1.2.3.1 and 2.3.4.1 respectively):
ip rule add from 1.2.3.4 table 234
ip route add default via 1.2.3.1 dev primary table 234
(the above should be optional)
ip rule add from 2.3.4.5 table 345
ip route add default via 2.3.4.1 dev mgmt table 345
Solution 2:
I found most of an answer here: https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming
More info here: http://linux-ip.net/html/routing-tables.html
Also, thanks to Tom Yan, for providing an answer that had most of the needed parts. But I'm posting this answer because none of the others is really complete, totally accurate, or easy to follow.
The following process works for me on Centos 7. I could not get it to work with NetworkManager, because NetworkManager does not read /etc/sysconfig/network-scripts/rule-* and route-*, and I could not find an equivalent nmcli command to make these changes persistent. If anyone has such a command, please share. So the first required step is to disable NetworkManager:
systemctl disable NetworkManager
systemctl enable network
systemctl stop NetworkManager
systemctl start network
Linux kernel 2.2 and 2.4 support multiple routing tables, each one numbered 0 to 255. The two routing tables normally employed are local (routing table 255) and main (routing table 254). These are listed in /etc/iproute2/rt_tables. You can create a new routing table by choosing an unused number (look in /etc/iproute2/rt_tables) and adding it to /etc/iproute2/rt_tables.
In my case, the "primary" interface is eth0, 192.168.20.20 with gateway 192.168.20.1, and the "mgmt" interface is eth1, 192.168.5.5 with gateway 192.168.5.1.
I am choosing the new routing table number 200, and new routing table name mgmt.
echo "200 mgmt" >> /etc/iproute2/rt_tables
Next, you need to create a rule to use the new routing table for traffic that was received on the mgmt interface. Other resources on the internet say you only need to do this for one interface, but you really need to do it on both, for the following reason: Notice, if you bring up only one interface, that interface is able to respond to ping, but if you bring up both interfaces, the one you brought up first responds to ping, and the second one doesn't. This means, if you bring up the mgmt interface first and then the primary interface, the primary will not respond. So it's good practice to define the rules on both interfaces.
This can be done one-time (non-persistent) as follows:
ip rule add from 192.168.20.20 table main
ip route add default via 192.168.20.1 dev eth0 table main
ip rule add from 192.168.5.5 table mgmt
ip route add default via 192.168.5.1 dev eth1 table mgmt
At this point, both interfaces should be responding. Now, to make it persistent, notice, if you read /etc/sysconfig/network-scripts/ifup-routes, it will parse all the rule-* and route-* files and pass each line as an argument to ip rule add or ip route add accordingly. So create four new files, as follows:
echo "from 192.168.20.20 table main" > /etc/sysconfig/network-scripts/rule-eth0
echo "default via 192.168.20.1 dev eth0 table main" > /etc/sysconfig/network-scripts/route-eth0
echo "from 192.168.5.5 table mgmt" > /etc/sysconfig/network-scripts/rule-eth1
echo "default via 192.168.5.1 dev eth1 table mgmt" > /etc/sysconfig/network-scripts/route-eth1
After reboot, you should find that both interfaces work by default.