How to make a linux VM working as a router

I have access to an openstack account where I can create Linux 14.04 VMs. I have created two network interfaces.

  1. "public-net" which is connected to the internet through a router

  2. "private-net" which is not exposed the internet

Now, I have created one VM, named "GATEWAY" which is connected to both the network interfaces and it has two internet address on eth0 (10.70.0.6) and eth1 (10.90.0.1). eth0 is exposed to the internet and eth1 is for the private network. The GATEWAY VM has a public ip-address on eth0.

Now I have created one more VM, named "AGENT" on private-net interface. ip address is 10.90.0.7 and make the default gateway as 10.90.0.1 (GATEWAY vm machine)

As the private VM is not exposed to any router so we can not have internet access to the VM. To enable internet access I have added a NAT rule on the GATEWAY vm as below:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

This will change the source address of all internet packets leaving the host GATEWAY as the address of the GATEWAY machine. Also, set the ipv4 packet forwarding=1 in the GATEWAY machine.

I can ping any external address from the GATEWAY machine but not from the internal agent machine. Not to mention that this private AGENT machine does not have internet access too.

Can anyone please help me set up the gateway VM such a way so that I can use it as a router and bring internet access to the private machines.

This is how my routing table looks like in AGENT machine:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.90.0.1       0.0.0.0         UG    0      0        0 eth0
10.90.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.169.254 10.90.0.2       255.255.255.255 UGH   0      0        0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

Here I am adding my tcpdump for icmp ping on both the intefaces.

eth1: interface connecting to private network.

18:43:39.309771 IP host-10-90-0-7.openstacklocal > 172.217.3.14: ICMP echo request, id 2395, seq 1, length 64
18:43:39.355430 IP 172.217.3.14 > host-10-90-0-7.openstacklocal: ICMP echo reply, id 2395, seq 1, length 64
18:43:40.318637 IP host-10-90-0-7.openstacklocal > 172.217.3.14: ICMP echo request, id 2395, seq 2, length 64
18:43:40.364178 IP 172.217.3.14 > host-10-90-0-7.openstacklocal: ICMP echo reply, id 2395, seq 2, length 64

eth0: interface connecting to the internet.

18:43:39.309796 IP host-10-70-0-6.openstacklocal > 172.217.3.14: ICMP echo request, id 2395, seq 1, length 64
18:43:39.355396 IP 172.217.3.14 > host-10-70-0-6.openstacklocal: ICMP echo reply, id 2395, seq 1, length 64
18:43:40.318679 IP host-10-70-0-6.openstacklocal > 172.217.3.14: ICMP echo request, id 2395, seq 2, length 64
18:43:40.364154 IP 172.217.3.14 > host-10-70-0-6.openstacklocal: ICMP echo reply, id 2395, seq 2, length 64
18:43:41.326618 IP host-10-70-0-6.openstacklocal > 172.217.3.14: ICMP echo request, id 2395, seq 3, length 64

Here I can see, ping respons is coming from the external address and its travelling both the intefaces. Even though it's being received by the eth1 to the private VM, its saying ping lost 100% packets.

---------     -------------------                                            -------------                                                      ------------
INTERNET |----| openstack-router| --10.70.0.1 --------10.70.0.6(NIC eth0) --| GATEWAY-VM |-- 10.90.0.1(NIC eth1) ---------10.90.0.7(NIC eth0) --| AGENT-VM |
---------     -------------------                                            -------------                                                      ------------ 

You are on the right track with your firewall rules. I do things the "old fashioned" way - write a script, put it in /etc and call it from /etc/rc.local. However you like to do it, here's what works for me.

The OS is Debian Jessie 64 bit via netinstall with only "standard system utilities" selected at tasksel time, running in VirtualBox on a Mint desktop. The eth0 is connecting to my LAN via a bridged interface and DHCP. The eth1 is the LAN side of a network of VM machines I use for experimenting. Copy/paste the firewall script to /etc/rc.firewall, make it executable, and call it in /etc/rc.local.

#!/bin/bash
# copyright me, licensed to you freely
# a very simple set of iptables commands 
# to allow forwarding between ethernet
# devices

# make sure forwarding is enabled in the kernel
echo 1 > /proc/sys/net/ipv4/ip_forward

# where is iptables located?
iptables=`which iptables`

# flush all existing rules
$iptables -F

# this is for NAT
# enable masquerading
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# don't forward packets from off-lan to lan if
# they are a brand new connection being initiated
$iptables -A FORWARD -i eth0 -o eth1 -m state --state NEW -j REJECT

# if the packets come from off-lan but they are
# related to a connection that was established from
# within the lan, go ahead and forward them
$iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT

# whatever traffic comes from the lan to go to
# the world allow thru
$iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

When all that is done and run, you should be able to see something like

root@router:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:e6:43:df  
          inet addr:192.168.1.126  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fee6:43df/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:382 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:61777 (60.3 KiB)  TX bytes:46468 (45.3 KiB)

root@router:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:af:50:e2  
          inet addr:10.99.99.1  Bcast:10.99.99.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feaf:50e2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:828 (828.0 B)

root@router:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
10.99.99.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@router:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             state NEW reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
root@router:~# cat /proc/sys/net/ipv4/ip_forward
1
root@router:~#