Can't chain more than one network namespace together
Problem Statement
With the below configuration a veth pair is created between the default/main net namespace and a netns called ns1.
The config also creates a second veth pair: veth2 is in netns ns1 and veth3 is in netns ns2, this joins ns1 to ns2 creating the chain: default netns-veth0 <-> veth1-ns1-veth2 <-> veth3-ns2.
sudo ip link add veth0 type veth peer name veth1
sudo ip -6 addr add CCFF::0/127 peer CCFF::1/127 dev veth0
sudo ip link set up dev veth0
sudo ip netns add ns1
sudo ip link set veth1 netns ns1
sudo ip -n ns1 -6 addr add CCFF::1/127 peer CCFF::0/127 dev veth1
sudo ip -n ns1 link set up dev veth1
sudo ip -n ns1 -6 route add default via CCFF::0
sudo ip link add veth2 type veth peer name veth3
sudo ip link set veth2 netns ns1
sudo ip -n ns1 -6 addr add CCFF::2/127 peer CCFF::3/127 dev veth2
sudo ip -n ns1 link set up dev veth2
sudo ip -n ns1 -6 route add CCFF::/64 via CCFF::3
sudo ip netns add ns2
sudo ip link set veth3 netns ns2
sudo ip -n ns2 -6 addr add CCFF::3/127 peer CCFF::2/127 dev veth3
sudo ip -n ns2 link set up dev veth3
sudo ip -n ns2 -6 route add default via CCFF::2
sudo ip -6 r add CCFF::/64 via CCFF::1
From the default netns I can ping veth0 which is in the same netns.
From the default netns I can ping veth1 and veth2 which are both in ns1.
From the default netns I can't pint veth3 which is in ns2.
If I extend the change as follows, by adding veth4 in ns2 and veth5 in ns3 I have the same problem. I can ping from any interface in ns1 to any interface in ns2, but can't reach any interface in ns3.
sudo ip link add veth4 type veth peer name veth5
sudo ip link set veth4 netns ns2
sudo ip netns exec ns2 ip -6 addr add CCFF::4/127 peer CCFF::5/127 dev veth4
sudo ip netns exec ns2 ip link set up dev veth4
sudo ip netns exec ns2 ip -6 route add CCFF::/64 via CCFF::5
sudo ip netns add ns3
sudo ip link set veth5 netns ns3
sudo ip netns exec ns3 ip -6 addr add CCFF::5/127 peer CCFF::4/127 dev veth5
sudo ip netns exec ns3 ip link set up dev veth5
sudo ip netns exec ns3 ip -6 route add default via CCFF::4
It seems I am only able to ping an interface in a directly "neighboring" / "connected" netns to the one I am pinging from. I am unable to ping through a chain of net namespaces. The routing is all valid; default netns can ping any interface in ns1 but nothing further, interfaces in ns1 can ping any interface in the default netns or ns2 but nothing in ns3, and ns3 can ping anything in ns2 but nothing beyond in ns1 or the default netns.
Is this a limitation of network namespacaes?
Troubleshooting
IPv6 forwarding is enabled, ip6tables is just set to "allow all", I'm not sure what else to check.
$ip -6 r
ccff::1 dev veth0 proto kernel metric 256 pref medium
ccff::/127 dev veth0 proto kernel metric 256 pref medium
ccff::/64 via ccff::1 dev veth0 metric 1024 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
$sudo ip -n ns1 -6 r
ccff:: dev veth1 proto kernel metric 256 pref medium
ccff::/127 dev veth1 proto kernel metric 256 pref medium
ccff::3 dev veth2 proto kernel metric 256 pref medium
ccff::2/127 dev veth2 proto kernel metric 256 pref medium
ccff::/64 via ccff::3 dev veth2 metric 1024 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth2 proto kernel metric 256 pref medium
default via ccff:: dev veth1 metric 1024 pref medium
$sudo ip -n ns2 -6 r
ccff::2 dev veth3 proto kernel metric 256 pref medium
ccff::2/127 dev veth3 proto kernel metric 256 pref medium
ccff::5 dev veth4 proto kernel metric 256 pref medium
ccff::4/127 dev veth4 proto kernel metric 256 pref medium
ccff::/64 via ccff::5 dev veth4 metric 1024 pref medium
fe80::/64 dev veth3 proto kernel metric 256 pref medium
fe80::/64 dev veth4 proto kernel metric 256 pref medium
default via ccff::2 dev veth3 metric 1024 pref medium
$sudo ip -n ns3 -6 r
ccff::4/127 dev veth5 proto kernel metric 256 linkdown pref medium
default via ccff::4 dev veth5 metric 1024 linkdown pref medium
$cat /proc/sys/net/ipv6/conf/all/forwarding
1
$cat /proc/sys/net/ipv6/conf/default/forwarding
1
$sudo ip6tables-save
# Generated by ip6tables-save v1.8.4 on Wed Nov 17 22:02:48 2021
*filter
:INPUT ACCEPT [76565:173401906]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [50440:6536664]
COMMIT
# Completed on Wed Nov 17 22:02:48 2021
$lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
$uname -a
Linux l13-ubuntu 5.11.0-40-generic #44~20.04.2-Ubuntu SMP Tue Oct 26 18:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
IPv6 forwarding is enabled
Probably not: When I tried this on my system with the commands given at the beginning of the question, I also cannot reach veth3:
$ ping -6 CCFF::3
PING CCFF::3(ccff::3) 56 data bytes
^C
--- CCFF::3 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12290ms
However, when I enable forwarding in ns1
echo 1 | sudo ip netns exec ns1 tee /proc/sys/net/ipv6/conf/all/forwarding
it works:
$ ping -6 CCFF::3
PING CCFF::3(ccff::3) 56 data bytes
64 bytes from ccff::3: icmp_seq=1 ttl=63 time=0.112 ms
64 bytes from ccff::3: icmp_seq=2 ttl=63 time=0.054 ms
So it's likely you also do not have IPv6 forwarding enabled in ns1. (And remember forwarding is per namespace; did you enable it in the main namespace, but not in ns1?)
I debugged this by doing a tcpdump on each interface; that the pings did not reach veth2 was a sign that forwarding was not enabled.
And if you are wondering "but why can I ping veth2 if forwarding is not enabled": Linux treats all local addresses the same, so you can reach any local address from any interface. This is completely unrelated to forwarding.
BTW, it helps a lot of you run an xterm in each namespace; that makes debugging a lot easier.
I came to the same totally valid conclusion as @dirkt did, each namespace is a network host by itself, including the setting whether it behaves like a router or not (which defaults to off). /proc/sys/net settings are separate for each namespace. resolving is separate, interface links are separate (but hostname isn't, for that you have to create UTS namespaces). So basically that is your answer.
Not to detract in anyway, but for future reference, I will add some things and rearrange and simplify this slightly. You don't actually need the routing setting in your default namespace, unless you would want to route outside traffic to these namespaces or vice versa.
To sum up a construction for default namespace and ns1 plus ns2.
# create namespaces
ip netns add ns1
ip netns add ns2
# loopbacks, nice to have
ip -n ns1 link set lo up
ip -n ns2 link set lo up
# make veth pairs for each new namespace
ip link add veth0 type veth peer name veth1
ip link add veth2 type veth peer name veth3
# add interfaces to their namespaces
ip link set veth1 netns ns1
ip link set veth2 netns ns1
ip link set veth3 netns ns2
# assign addresses
ip -6 addr add CCFF::0/127 dev veth0
ip -n ns1 -6 addr add CCFF::1/127 dev veth1
ip -n ns1 -6 addr add CCFF::2/127 dev veth2
ip -n ns2 -6 addr add CCFF::3/127 dev veth3
# set the new links to up
ip link set up dev veth0
ip -n ns1 link set up dev veth1
ip -n ns1 link set up dev veth2
ip -n ns2 link set up dev veth3
# namespaces that should forward
ip netns exec ns1 sysctl -w net.ipv6.conf.all.forwarding=1
# route any ip6 outward towards host through namespace 1
ip -n ns2 -6 route add default via CCFF::2
# route any ccff ip6 inward from host
# to a more inner space through namespace 1
ip -6 r add CCFF::/64 via CCFF::1
Some tests you can then do:
ip netns list
ip netns exec ns2 ping6 ccff::0
ip netns exec ns2 ping6 ccff::1
ip netns exec ns2 ping6 ccff::3
# Or put your shell (e.g. bash) in the namespace ns2
ip netns exec ns2 /bin/bash
ping6 ccff::0
exit
You can also do some resolving and hosts file changes
# Setup a resolver
# (replace with your own DNS, does not work with a loopback resolver)
mkdir -p /etc/netns/ns2
echo nameserver dns-ip > /etc/netns/ns2/resolv.conf
# Maybe give it its own hosts file, to do edits
cp /etc/hosts /etc/netns/ns2/hosts