Second ENI in AWS VPC is not accessible on Ubuntu instance

I'm just getting into VPC, trying to understand how everything works. So far the biggest hurdle I've run into is that any time I add a second Elastic NIC to a machine, that second IP is not accessible by any others in the VPC. Here's what I did

• Launched Canonical provided AMI for Ubuntu 12.10 x64 EBS.
• During launch I configured it for two network interfaces (same subnet)
• Once the machine was up, I added the following to /etc/network/interfaces :

auto eth1

iface eth1 inet dhcp

• ifup eth1
• Run ifconfig, verify the second address is up.

On my primary (internet accessible) instance:

• ping (IP for new instance eth0) - Works
• ping (IP for new instance eth1) - FAILS

There are no ACL's that prevent ping, as it works with eth0. There is no firewall setup on the machine. I've tried 4 different instances across several SGs and AZs with multiple interfaces, all with the same result.

I've been bashing my head against the wall for longer than I care to admit on this. I cannot figure out where the error is on this.

The routing table by default will only route traffic to eth0. Even though ubuntu detects the other ENI, you still have to route traffic to it.

You'll have to do some advanced routing:

1) Enabling access to 2nd ENI immediately and temporarily.

source: http://www.rjsystems.nl/en/2100-adv-routing.php

# this will show your route table, i'll assume you have eth0 and eth1
# and your default is for eth0 to point to the gateway
# for this example lets assume the following:
# eth0 = 192.168.100.5 
# eth1 = 192.168.100.10
# gateway = 192.168.100.1
ip route show ;

# first step is to create a routing table for your new device
cat /etc/iproute2/rt_tables ;
echo 2 eth1_rt >> /etc/iproute2/rt_tables ;

# next add the eth1_rt route table, so by default it will also point to the gateway
ip route add default via 192.168.100.1 dev eth1 table eth1_rt ;

# next take a look at your ip rules
# i'll assume the defaults here, and things flows to default with priority 32767
ip rule;

# let's add a rule, if we see traffic from eth1's IP address,
# use its new routing table we setup, and give it higher priority than default
ip rule add from 192.168.100.10 lookup eth1_rt prio 1000 ;

# done! now check your traffic from both IPs, they should both work.

2) Enabling access to 2nd ENI on reboot but persistently.

source: http://blog.bluemalkin.net/multiple-ips-and-enis-on-ec2-in-a-vpc/

Additionally, if you want this change to persist, you can make all these changes in the interface file and just restart the network service or reboot for it to take effect.

# NOTE: add the eth1_rt routing table to /etc/iproute2/rt_tables as show in previous section

# original config to make dchp, I add mine to /etc/network/interfaces.d/eth1.cfg
auto eth1
iface eth1 inet dchp
    # your extra rules for eth1
    up ip route add default via 192.168.100.1 dev eth1 table eth1_rt
    up ip rule add from 192.168.100.10 lookup eth1_rt prio 1000

For this to take full effect, reboot the system.

NOTE: I tried /etc/init.d/networking restart; but it didn't pick up the route/rule changes, not sure why, so I had reboot. In the event you want to make it immediate and persistent, do both methods.