Configuring multiple Ethernet connections in a linux server

I have the following topology

                                                                                        .───────────────.        
   ┌──────────────────┐                                                             _.─'                 `──.    
   │                  │                                                          ,─'                         '─. 
   │    ISP Router    │ Dynamic IP                                              ;                               :
   │     with NAT     ├─────────────────────────────────────────────────────────:           Internet            ;
   │                  │                                                          ╲                             ╱ 
   └─────────┬────────┘                                                           '─.                       ,─'  
             │                                                                       `──.               _.─'     
             │  10.0.0.1                                                                 `──────┬──────'         
             │                                                                                  │                
      .──────┴────.                                                                       .─────┴─────.          
   ,─'             '─.                   ┌─────────────────────────┐                   ,─'             '─.       
 ,'                   `.                 │                         │                 ,'                   `.     
;          LAN          :  Static DHCP┌──┴───┐                  ┌──┴───┐ Static IP  ;      Public Net       :    
:                       ;─────────────┤ eth1 │     My Server    │ eth0 ├────────────:                       ;    
 ╲     10.0.0.0/24     ╱     10.0.0.5 └──┬───┘                  └──┬───┘  1.2.3.5    ╲     1.2.3.4/20      ╱     
  `.                 ,'                  │       Debian Buster     │    mydomain.com  `.                 ,'      
    '─.           ,─'                    │                         │                    '─.           ,─'        
       `─────────'                       └─────────────────────────┘                       `─────────'

Because of historical reasons, my ISP uses a separate network to provide static IPs. This means I have to connect my server directly to a special connection that is directly exposed to the internet (in the diagram this is eth0 with IP 1.2.3.5).

However, I use this same server as a NAS in my home and since that's a separate network altogether I connect it using a different interface eth1 and create a static DHCP allocation with address 10.0.0.5. This way, other devices in the home network can find it and access the network mounts.

Lastly, I have a DNS record with my domain, in this example mydomain.com, pointing to 1.2.3.5.

I'm running into a couple of issues:

  • When I try to access mydomain.com from my LAN it doesn't work. I believe this is because the Server is getting the packet from interface eth0 but then replying through eth1 and the other device is dropping the response since it's from a different IP.
  • On boot the Server should pick 1.2.3.4 as the network to direct internet traffic to so external access works through mydomain.com. However, since both eth0 and eth1 have internet access through the respective gateways, it sometimes boots and picks 10.0.0.1 as the default gateway and then accessing mydomain.com doesn't work. I believe this is a similar issue with traffic entering through eth0 but leaving through eth1.

While I'm familiar with basic networking stuff, I don't have experience with having a server connected to multiple networks at once like this. Ideally I'd like to configure the server so that:

  • All internet traffic exits through eth0
  • All traffic from entering eth0 is replied through eth0
  • All traffic from entering eth1 is replied through eth1

My server runs Debian, I'd appreciate any help.

EDIT: So after disabling the eth1 gateway and playing a bit with tcpdump I was able to confirm that traffic is entering from eth0 and leaving eth1. The issue is that the ISP is not NATing traffic going to the endpoint 1.2.3.5 and then the server is identifying the 10.0.0.X addresses as located on interface eth1.


This is a routing issue which can be solved with source-based routing or policy routing. This involves two routing tables, one for each source address:

See http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html for more details.