Openvpn server not forwarding ping traffic from tun0 to eth0 for rest of the hosts in the subnet

Currently I have a openvpn server and client setup with rounting (not bridging)

When I try to ping from my client to server ip address it works fine. But when I try to ping rest of the subnet hosts behind openvpn server, it does not work. Can someone spot something obviously wrong in my setup. (openvpn server is on 10.10.145.181 and host is on 10.10.146.8 ip address. They are in two separate subnets. I can ping 10.10.146.8 from 10.10.145.181 directly by sshing into that host. Only when I go via VPN it does not work.)

From what I understand, the ping traffic makes it to vpn server on tun0 interface, but then vpn server is not forwarding it over eth0 to appropriate host & hence the pinged host does not see any traffic & packet is dropped. But what could be causing this? Is there a setting in openvpn to forward traffic from tun0 to eth0?

Here is what i observe...

On openvpn client host:

> ping 10.10.146.8
PING 10.10.146.8 (10.10.146.8) 56(84) bytes of data.
<no further output>

On openvpn server host:

> sudo tcpdump -i tun0 'icmp[icmptype] = icmp-echo or icmp[icmptype] = icmp-echoreply'
00:34:32.624639 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1863, length 64
00:34:33.634564 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1864, length 64
00:34:34.640753 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1865, length 64
00:34:35.648922 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1866, length 64
00:34:36.659062 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1867, length 64
00:34:37.665402 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1868, length 64
00:34:38.673295 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1869, length 64
00:34:39.685336 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1870, length 64
00:34:40.687703 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1871, length 64
00:34:41.695766 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 1644, seq 1872, length 64
> sudo tcpdump -i eth0 'icmp[icmptype] = icmp-echo or icmp[icmptype] = icmp-echoreply'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:14:48.583673 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 4807, seq 442, length 64
04:14:49.592908 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 4807, seq 443, length 64
04:14:50.600010 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 4807, seq 444, length 64
04:14:51.616401 IP 10.10.0.6 > 10.10.146.8: ICMP echo request, id 4807, seq 445, length 64

On the host in the subnet which I am trying to ping(10.10.146.8):

> sudo tcpdump -i eth0 'icmp[icmptype] = icmp-echo or icmp[icmptype] = icmp-echoreply'
sudo: unable to resolve host ip-10-10-146-8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
<no further output>

The syslog(openvpn log) is saying:

Jul  8 00:36:25 ip-10-10-145-181 ovpn-server[1513]: xyz/<ip_address>:35315 UDPv4 READ [133] from [AF_INET]<ip_address>:35315: P_DATA_V1 kid=0 DATA len=132 
Jul  8 00:36:25 ip-10-10-145-181 ovpn-server[1513]: xyz/<ip_address>:35315 TUN WRITE [84]

netstat on openvpn client:

10.10.0.1       10.10.0.5       255.255.255.255 UGH       0 0          0 tun0
10.10.0.5       0.0.0.0         255.255.255.255 UH        0 0          0 tun0
10.10.146.0     10.10.0.5       255.255.255.0   UG        0 0          0 tun0

netstat on openvpn server:

10.10.0.0       10.10.0.2       255.255.255.0   UG        0 0          0 tun0
10.10.0.2       0.0.0.0         255.255.255.255 UH        0 0          0 tun0
10.10.145.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

I have a statement in openvpn server config to route the traffic from client side & I do see that happening.

push "route 10.10.146.0 255.255.255.0"

Additional information for Andrew's question

> echo "sysctl -a | grep 'forwarding = 1'" | sudo -s
error: permission denied on key 'vm.compact_memory'
error: permission denied on key 'net.ipv4.route.flush'
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.tun0.forwarding = 1
error: permission denied on key 'net.ipv6.route.flush'



> sudo iptables -L INPUT
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         


> sudo iptables -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Update: Actually now I see traffic on eth0 on server, but that traffic is not making it to the network and being received by the other host. I think this is Amazon VPC problem.

Solution 1:

Ok, I figured this out after a few hours of investigation.

The problem is with the forwarding setup. The packets forwarded to eth0 port does not have correct source ip address of the host within the network. The ip addressis from VPN.

05:07:43.991961 IP 10.8.0.6 > 10.10.146.8: ICMP echo request, id 3497, seq 499, length 64

You can switch that by enabling equivalent of NAT (on routers) in linux OS:

iptables -t nat -A POSTROUTING -o <eth0 or whatever else> -j MASQUERADE

This fixed the issue for me.