ip routing for extra eni on RHEL ec2 instances
I have noticed that when one add an additional ENI on an ec2, getting routing to work is so difficult and there is no documentation anywhere to refer to.
I do know that if one uses the Amazon Linux ami, everything works well automatically without any extra work. But using any other ami, one has to configure the routing to work.
non-Amazon Linux
[root@ip-10-11-2-20 ~]# cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Red Hat Enterprise Linux Server release 7.3 (Maipo)
[root@ip-10-11-2-20 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.11.2.1 0.0.0.0 UG 100 0 0 eth0
0.0.0.0 10.11.2.1 0.0.0.0 UG 101 0 0 eth1
10.11.2.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
10.11.2.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1
[root@ip-10-11-2-20 ~]# ip route
default via 10.11.2.1 dev eth0 proto static metric 100
default via 10.11.2.1 dev eth1 proto static metric 101
10.11.2.0/24 dev eth0 proto kernel scope link src 10.11.2.20 metric 100
10.11.2.0/24 dev eth1 proto kernel scope link src 10.11.2.247 metric 101
[root@ip-10-11-2-20 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 10.11.2.20 netmask 255.255.255.0 broadcast 10.11.2.255
inet6 fe80::1012:30ff:feca:4bd0 prefixlen 64 scopeid 0x20<link>
ether 12:12:30:ca:4b:d0 txqueuelen 1000 (Ethernet)
RX packets 3065 bytes 258041 (251.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2375 bytes 351169 (342.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 10.11.2.247 netmask 255.255.255.0 broadcast 10.11.2.255
inet6 fe80::1034:1bff:fe8d:c742 prefixlen 64 scopeid 0x20<link>
ether 12:34:1b:8d:c7:42 txqueuelen 1000 (Ethernet)
RX packets 473 bytes 22008 (21.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 58 bytes 5544 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 140 bytes 11924 (11.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 140 bytes 11924 (11.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.20
PING 10.11.2.20 (10.11.2.20) 56(84) bytes of data.
64 bytes from 10.11.2.20: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 10.11.2.20: icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from 10.11.2.20: icmp_seq=3 ttl=64 time=0.025 ms
--- 10.11.2.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.013/0.021/0.027/0.008 ms
[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.247
PING 10.11.2.247 (10.11.2.247) 56(84) bytes of data.
64 bytes from 10.11.2.247: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 10.11.2.247: icmp_seq=2 ttl=64 time=0.024 ms
64 bytes from 10.11.2.247: icmp_seq=3 ttl=64 time=0.024 ms
--- 10.11.2.247 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.013/0.020/0.024/0.006 ms
Amazon Linux
[root@ip-10-11-2-149 ~]# cat /etc/*release
NAME="Amazon Linux AMI"
VERSION="2017.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.03"
PRETTY_NAME="Amazon Linux AMI 2017.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2017.03
[root@ip-10-11-2-149 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.11.2.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.11.2.1 0.0.0.0 UG 10001 0 0 eth1
10.11.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.11.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
[root@ip-10-11-2-149 ~]# ip route
default via 10.11.2.1 dev eth0
default via 10.11.2.1 dev eth1 metric 10001
10.11.2.0/24 dev eth0 proto kernel scope link src 10.11.2.149
10.11.2.0/24 dev eth1 proto kernel scope link src 10.11.2.61
169.254.169.254 dev eth0
[root@ip-10-11-2-149 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 12:B2:B8:77:D0:F8
inet addr:10.11.2.149 Bcast:10.11.2.255 Mask:255.255.255.0
inet6 addr: fe80::10b2:b8ff:fe77:d0f8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:1490 errors:0 dropped:0 overruns:0 frame:0
TX packets:1441 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:129285 (126.2 KiB) TX bytes:143368 (140.0 KiB)
eth1 Link encap:Ethernet HWaddr 12:9B:45:32:EB:BA
inet addr:10.11.2.61 Bcast:10.11.2.255 Mask:255.255.255.0
inet6 addr: fe80::109b:45ff:fe32:ebba/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:263 errors:0 dropped:0 overruns:0 frame:0
TX packets:347 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:16825 (16.4 KiB) TX bytes:25705 (25.1 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:140 (140.0 b) TX bytes:140 (140.0 b)
[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.149
PING 10.11.2.149 (10.11.2.149) 56(84) bytes of data.
64 bytes from 10.11.2.149: icmp_seq=1 ttl=255 time=0.018 ms
64 bytes from 10.11.2.149: icmp_seq=2 ttl=255 time=0.027 ms
64 bytes from 10.11.2.149: icmp_seq=3 ttl=255 time=0.026 ms
--- 10.11.2.149 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2035ms
rtt min/avg/max/mdev = 0.018/0.023/0.027/0.006 ms
[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.61
PING 10.11.2.61 (10.11.2.61) 56(84) bytes of data.
64 bytes from 10.11.2.61: icmp_seq=1 ttl=255 time=0.018 ms
64 bytes from 10.11.2.61: icmp_seq=2 ttl=255 time=0.027 ms
64 bytes from 10.11.2.61: icmp_seq=3 ttl=255 time=0.029 ms
--- 10.11.2.61 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms
rtt min/avg/max/mdev = 0.018/0.024/0.029/0.007 ms
Now here is the problem BELOW...see how the non-mazon Linux can ping both eth0 and eth1 ENIs on the Amazon Linux but the Amazon Linux cannot ping the eth1 of the non-Amazon Linux
INTER NETWORKING
non-Amazon Linux
[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.149
PING 10.11.2.149 (10.11.2.149) 56(84) bytes of data.
64 bytes from 10.11.2.149: icmp_seq=1 ttl=255 time=0.629 ms
64 bytes from 10.11.2.149: icmp_seq=2 ttl=255 time=0.486 ms
64 bytes from 10.11.2.149: icmp_seq=3 ttl=255 time=0.472 ms
--- 10.11.2.149 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.472/0.529/0.629/0.070 ms
[root@ip-10-11-2-20 ~]# ping -c 3 10.11.2.61
PING 10.11.2.61 (10.11.2.61) 56(84) bytes of data.
64 bytes from 10.11.2.61: icmp_seq=1 ttl=255 time=0.595 ms
64 bytes from 10.11.2.61: icmp_seq=2 ttl=255 time=0.560 ms
64 bytes from 10.11.2.61: icmp_seq=3 ttl=255 time=0.522 ms
--- 10.11.2.61 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.522/0.559/0.595/0.029 ms
Amazon Linux
[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.20
PING 10.11.2.20 (10.11.2.20) 56(84) bytes of data.
64 bytes from 10.11.2.20: icmp_seq=1 ttl=64 time=0.446 ms
64 bytes from 10.11.2.20: icmp_seq=2 ttl=64 time=0.465 ms
64 bytes from 10.11.2.20: icmp_seq=3 ttl=64 time=0.481 ms
--- 10.11.2.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2030ms
rtt min/avg/max/mdev = 0.446/0.464/0.481/0.014 ms
[root@ip-10-11-2-149 ~]# ping -c 3 10.11.2.247
PING 10.11.2.247 (10.11.2.247) 56(84) bytes of data.
^C
--- 10.11.2.247 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2028ms
How can i allow the routing to the eth1 extra ENI added to the RHEL 7 ec2 instance?
Solution 1:
Just to add exactly what i did and things started to work
Just added the following 4 lines from the URL @strongjz mentioned
non-Amazon Linux
ip route add default via 10.11.2.1 dev eth0 tab 1
ip route add default via 10.11.2.1 dev eth1 tab 2
ip rule add from 10.11.2.20/32 tab 1 priority 500
ip rule add from 10.11.2.247/32 tab 2 priority 600
And the rest was history..all ping works to all interfaces, both eth0 and eth1!!!
All thanks to this URL https://forums.aws.amazon.com/message.jspa?messageID=404398