Linux bridge network influenced by route table

I was doing an experiment about linux bridge and my network topology is like:

enter image description here

As you can see, there are two hosts located in a LAN, Host1(10.74.68.58) and Host2(10.74.68.47). On Host1, I created a bridge br0 and assigned an IP for it (192.168.3.101). Then I attached eth0 to the bridge:

[[email protected]:~] # bridge link
2: eth0 state UP : <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2

I set the default route as br0 and it is ok to ping 10.74.68.47:

[[email protected]:~] # ip r
default dev br0  scope link
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
192.168.3.0/24 dev br0  proto kernel  scope link  src 192.168.3.101

But things became unexplainable when I set default route to eth0 :

[[email protected]:~] # ip r
default dev eth0  scope link
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
192.168.3.0/24 dev br0  proto kernel  scope link  src 192.168.3.101

When eth0 is the default route interface, I try to ping host2 in two different ways:

1, ping 10.74.68.47

Failed. After I checked tcpdump file (captured on interface br0), I found that br0 only received ARP response. So there is no ARP info on interface eth0, thus it can not get the mac of host2. I think this is the right behaviour, is my understanding right?

2, Then I tried ping -I br0 10.74.68.47

I wanted to use -I option to steer clear of default route, but also failed. After I check the tcpdump file (captured on interface br0), I found there is already a pair of icmp echo request and echo reply packet. This confused me a lot. Now that br0 have received echo reply, why can't I ping to host2 successfully?

[[email protected]:~] # ping -I br0 10.74.68.47
2 packets transmitted, 0 received, 100% packet loss, time 1006ms

enter image description here

Can you guys give me some pointers?


Solution 1:

Bridging doesn't work the way you think it works. :-)

A bridge is only concerned with OSI level 2 (Ethernet frames). On this level, IP addresses etc. don't exist. Conceptually, you can think of a bridge as a collection of ethernet interfaces. Each interface is called a port, and a packet that goes into one port comes out on all other ports. (Actually, in the Linux implementation, there's an optimization that keeps a table of seen MAC addresses, but conceptually, it doesn't matter).

So a bridge can connect ("bridge") several ethernet segments into one big segment.

Then what does it mean to "give a Linux bridge an IP address"? In the Linux implementation, the bridge is not a separate hardware device (like they originally were), it's also accessible from the host itself. That means it acts like a kind of "super-ethernet interface" with many ports, but packets that go into the kernel or out from this kernel to or from any of these ports reach the Linux OS under a single IP address.

So as soon as you make an ethernet interface a slave (port) of a bridge, it ceases to have its own address. The only thing that counts is the IP address of the bridge.

In other words, making a bridge with only a single port makes no sense (you could have used the interface by itself). Trying to route packets to a port of a bridge makes no sense (as far as the kernel is concerned, the bridge is a single device).

If you want to play around with a bridge, you need a structure like this:

  10.0.2.1/23    10.0.2.2/23    10.0.3.254/23     10.0.3.1/23    10.0.3.2/23 

  ............   ............   ...............   ............   ............
  .  Host A  .   .  Host B  .   .  Host X     .   .  Host C  .   .  Host D  .
  .          .   .          .   . <-- br0 --> .   .          .   .          .
  .   eth0   .   .   eth0   .   . eth0   eth1 .   .   eth0   .   .   eth0   .
  .....|......   .....|......   ...|......|....   .....|......   .....|......
       |              |            |      |            |              |      
  -----+--------------+------------+      +------------+--------------+------

  <-------- left Segment  --------->      <------- right Segment ----------->

Here the left segment with hosts A and B is bridged by host X to the right segment with hosts C and D, and each host is accessible by a single IP address (which is assigned to interfaces or the bridge as a whole).