network level of veth doesn't respond to arp

I decided to play with veth : create a veth pair and send a ping from one end to another.

$ ip link add type veth
$ ip addr add 192.168.99.1 dev veth4
$ ip addr add 192.168.99.2 dev veth5
$ ip link dev veth4 set up
$ ip link dev veth5 set up

Let's check.

$ ip a

18: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 6a:dc:02:5b:f0:f3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global veth4
       valid_lft forever preferred_lft forever
19: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 22:ec:d5:e8:7c:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.2/24 scope global veth5
       valid_lft forever preferred_lft forever

Everything seems to be ok. Now try to ping.

$ ping -I veth4 192.168.99.2
PING 192.168.99.2 (192.168.99.2) from 192.168.99.1 veth4: 56(84) bytes of data.
From 192.168.99.1 icmp_seq=1 Destination Host Unreachable
From 192.168.99.1 icmp_seq=2 Destination Host Unreachable
From 192.168.99.1 icmp_seq=3 Destination Host Unreachable


$ sudo tshark -i veth5
Capturing on 'veth5'
l  1   0.000000 6a:dc:02:5b:f0:f3 -> Broadcast    ARP 42 Who has 192.168.99.2?  Tell 192.168.99.1
1   2   1.003206 6a:dc:02:5b:f0:f3 -> Broadcast    ARP 42 Who has 192.168.99.2?  Tell 192.168.99.1

So veth5 recevies arp requests but doesn't bother to answer. What's the matter?

This is a very complex, very old question, involving several different areas, specifically VETH, Network Namespaces, ARP, route tables, and NAT...

I am adding the answer to this problem not just for OP, but for others out there, hopefully saving someone else from the grey hairs this gave me...

Suffice to say, after much research and testing -- veth pairs can exactly be used this way, as they are literally two seperate virtual interfaces. However, because they are virtual -- you will encounter some really odd problems with regards to ARP tables/entries. After several weeks of tackling this exact scenario (necessary for work), I have figured out exactly how to do this, and in the process have learned way more about ARP, NAT, and routing than I wanted to.

In order to accomplish exactly what you want, type the following (in my case, Ubuntu 16.04). Please note that once you are in the test namespace, you have to exit bash to leave, as everything (route tables, iptables, etc) from that point on is different than on the host machine. I usually open up two terminals, one that stays in the host -- the other that stays in the guest namespace.

Preface

# Tell the system we want to support IPv4 forwarding  
add/verify that the following is in `/etc/sysctl.conf`

`net.ipv4.ip_forward = 1`

# if it wasn't there, reparse syscrtl + restart networking
sysctl -p;
/etc/init.d/networking restart;

HOST SETUP

# create a veth pair
ip link add name vHOST type veth peer name vGUEST

# choose a private MAC address and private IP address
ifconfig vHOST hw ether 02:1d:8d:dd:0c:61
ifconfig vHOST 10.11.0.1/24 up

# have to setup routes   FROM HOST -> GUEST
ip route add 10.111.0.0/24 via 10.11.0.1 dev vHOST  # gateway

# have to explicitly assign what the MAC is for the vGUEST in the vHOST interface ARP table
arp -i vHOST -s 10.111.0.1 02:1d:8d:dd:0c:60 # vGUEST ip + mac address

# We must tell vHOST-vGUEST "tunnel", vHOST side that it's not a REAL bridge, so any ARP requests must be answered by vHOST
echo 1 > /proc/sys/net/ipv4/conf/vHOST/proxy_arp


# setup forwarding + NAT (so packets can come back)
iptables -A FORWARD -s 10.111.0.0/24 -o vHOST -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.111.0.1 -j SNAT --to 192.168.42.124

GUEST SETUP

# create the "test" namespace, so we can verfiy settings
ip netns add test

# add vGUEST "interface" to the "test" namespace
ip link set vGUEST netns test

# enter the "test" namespace with a bash shell
ip netns exec test bash

# choose a private MAC address and private IP address
ifconfig vGUEST hw ether 02:1d:8d:dd:0c:60
ifconfig vGUEST 10.111.0.1/24 up

# have to setup routes   FROM GUEST -> HOST
ip route add default via 10.111.0.1 dev vGUEST   # gateway

# have to explicitly assign what the MAC is for the vHOST in the vGUEST interface ARP table
arp -i vGUEST -s 10.11.0.1 02:1d:8d:dd:0c:61 # vHOST ip + mac address

now, GUEST/HOST can ping eachother, and vGUEST can ping outside the machine

Of paticular note, multimac can also do the exact same thing as veth, in a slightly different way, as multimac supports a many-to-one relationship, where veth is a one-to-one relationship.

In order to support this on without namespaces, you have to use policy routing (which is a bit more complex than network namespaces)

Veth pairs aren't used this way. A veth pair is literally a device pipe, one end of the pipes packets come out the other end.

The simplest synonym I can offer is imagining one half of the pair is a ethernet device, whilst the other end is a switch port the device is plugged into. You shouldn't treat the pair as being two separate devices, but one device which has two 'ends' to push/pull from.