Wireguard mesh between public and local network
If nodes 4,5,6 are each behind their own CGNAT, then as usual, they can't manage to do proper communication between them. They have to rely on nodes 1,2,3 to relay traffic. So these CGNAT nodes will require extra AllowedIPs for some public peers in their settings to allow other CGNATs nodes through the public nodes, and the public nodes must be set as routers.
Example for node 4 assumed to have IP address 10.1.0.51/32 and using node 2 as routing node:
[Interface]
Address = 10.1.0.51/32
PrivateKey =
[Peer] # example public node 1
PublicKey =
AllowedIPs = 10.1.0.1/32
Endpoint = X.X.X.X:5888
PersistentKeepalive = 25
[Peer] # example public node 2
PublicKey =
AllowedIPs = 10.1.0.2/32,10.1.0.52/32,10.1.0.53/32
Endpoint = Y.Y.Y.Y:5888
PersistentKeepalive = 25
[Peer] # example public node 3
PublicKey =
AllowedIPs = 10.1.0.3/32
Endpoint = Z.Z.Z.Z:5888
PersistentKeepalive = 25
Other CGNAT nodes' IP addresses are expected to be routed through node 2 and should be added to that peer's AllowedIPs. This should also add them automatically to the routing table.
Node 2 must now also be a router, at least on its WireGuard interface which will be both ingress and egress. On Linux this would be done for example with (assuming an interface wg0):
sysctl -w net.ipv4.conf.wg0.forwarding=1
Its firewall rules must also allow forwarded traffic on wg0.
Please note that there's no need to define the other CGNAT peers on the CGNAT nodes, and if defined they must not have other CGNAT nodes' IP addresses in AllowedIPs:
- they won't be reachable directly and,
- if the same address is set in subsequent AllowedIPs it will be deleted from each previous definition and only the last defined peer will have it. With cryptokey routing IP address(es) <=> peer.
Nodes 5 and 6 must have a compatible configuration (also using node 2 as router). You could also imagine having instead:
- split roles where 4 and 5 are routed by node 2, 4 and 6 by node 3 and 5 and 6 by node 1,
- or having multiple IP addresses within different IP blocks so each block can be routed through a different routing node,
- or using onion routing with WireGuard traffic within WireGuard traffic so that the routing nodes can't decode traffic not intended for them,
- or using multiple independent WireGuard interfaces (which aren't subject to cryptokey interaction between themselves, but only to the routing table),
- or some combination of previous options,
[...]
In all cases, for any topology change even if due to a failure rather than an intended change, a way to synchronize configuration changes in all affected peers is needed and there's currently no dedicated tool to do this.
As a conclusion, here's a blog where BGP (that would be the missing tool) is used along WireGuard, with multiple addresses and also one interface per peer node with only that peer defined to sidestep cryptokey routing. There's probably something to be learnt from this but the topic is way too advanced for me.
Route-based VPN on Linux with WireGuard