Chaining WireGuard Servers: Can ping both from client, but can't access internet. IP routing issue?

Solution 1:

The kind folks over at #wireguard IRC channel on Libera.Chat helped me out!

The issues with my config. were as follows:

Server1's config has Server2 with AllowedIPs of just Server2's IP addresses. That wont permit "The Internet". It needs to be 0.0.0.0/0, ::0/0.
Server1's WireGuard interface must be configured to add routes (for all entries in AllowedIPs) to a custom IP routing table (let's call it wireguard2x) instead of the main table. Then add an IP policy rule that says that traffic with the input interface (iif) wg0 go to the custom table—which also means other kinds of traffic take the default route, per usual.

Corrected Configuration

Server2 Configuration

File: /etc/wireguard/wg0.conf

# Server2

[Interface]
PrivateKey = SERVER2_PRIVATE_KEY
Address = 10.103.213.2/24, fd6f:9403:2887:9cd6:10:103:213:2/112
ListenPort = 53701
SaveConfig = false

# CLIENTS

[Peer] # Server1
PublicKey = SERVER1_PUBLIC_KEY
PresharedKey = SERVER1_PRESHARED_KEY
# ↓ to allow traffic from client (10.103.213.11/32) via Server1 (10.103.213.1/32), allow both
AllowedIPs = 10.103.213.0/24, fd6f:9403:2887:9cd6:10:103:213:0/112

Firewall config. commands:

ufw allow 53701/udp comment 'WireGuard VPN'

iptables -A FORWARD -i wg0 -j ACCEPT &&
iptables -A FORWARD -o wg0 -j ACCEPT &&
ip6tables -A FORWARD -i wg0 -j ACCEPT &&
ip6tables -A FORWARD -o wg0 -j ACCEPT

iptables -t nat -A POSTROUTING -s 10.103.213.0/24 -o enp7s0 -j MASQUERADE
ip6tables -t nat -A POSTROUTING -s fd6f:9403:2887:9cd6:10:103:213:0/112 -o enp7s0 -j MASQUERADE

Server1 Configuration

File: /etc/wireguard/wg0.conf

# Server1

[Interface]
PrivateKey = SERVER1_PRIVATE_KEY
Address = 10.103.213.1/32, fd6f:9403:2887:9cd6:10:103:213:1/128
ListenPort = 53701
Table = wireguard2x
# ↓ should only be set if resolvconf or openresolv is installed on the system, otherwise let the system use defaults
# ↓ is unncessary if local DNS resolution is set up
#DNS = 1.1.1.1, 1.0.0.1, 2606:4700:4700::1111, 2606:4700:4700::1001
DNS = 10.103.213.1, fd6f:9403:2887:9cd6:10:103:213:1
SaveConfig = false

# CLIENTS

[Peer] # Server2
PublicKey = SERVER2_PUBLIC_KEY
PresharedKey = SERVER1_PRESHARED_KEY
Endpoint = SERVER2_PUBLIC_IP:53701
AllowedIPs = 0.0.0.0/0, ::0/0
#PersistentKeepalive = 25

[Peer] # PC
PublicKey = CLIENT_PUBLIC_KEY
PresharedKey = CLIENT_PRESHARED_KEY
AllowedIPs = 10.103.213.11/32, fd6f:9403:2887:9cd6:10:103:213:11/128

Firewall config. commands:

ufw allow 53701/udp comment 'WireGuard VPN'

iptables -A FORWARD -i wg0 -j ACCEPT &&
iptables -A FORWARD -o wg0 -j ACCEPT &&
ip6tables -A FORWARD -i wg0 -j ACCEPT &&
ip6tables -A FORWARD -o wg0 -j ACCEPT

echo 123 wireguard2x >> /etc/iproute2/rt_tables
ip rule add iif wg0 table wireguard2x

Client Configuration

# CLIENT: PC

[Interface]
PrivateKey = CLIENT_PRIVATE_KEY
Address = 10.103.213.11/32, fd6f:9403:2887:9cd6:10:103:213:11/128
DNS = 10.103.213.1, fd6f:9403:2887:9cd6:10:103:213:1

[Peer] # Server1
PublicKey = SERVER1_PUBLIC_KEY
PresharedKey = CLIENT_PRESHARED_KEY
Endpoint = SERVER1_PUBLIC_IP:53701
AllowedIPs = 0.0.0.0/0, ::0/0
#PersistentKeepalive = 25

NOTES:

You can see all the policy routing rules that are currently in effect using this command: ip rule list or ip rule
View the routing tables with, for e.g., ip route show table wireguard2x or ip route list table wireguard2x.
To flush the route cache: ip route flush cache
Further reading: Linux Advanced Routing & Traffic Control (LARTC) HOWTO
You can monitor Client's network traffic flow—if on Linux, using sudo iptraf-ng; on macOS using sudo iftop. (To be run on the client.)
- I found running that hard to parse due to too many live changes. So I was suggested trying route get 192.0.2.0 (which shows the interface being used, e.g., interface: utun2) and then running ifconfig <interface> (e.g., ifconfig utun2) on macOS. The latter should show the Client's private (WireGuard peer) IP address, e.g., 10.103.213.11, confirming that the traffic is being routed through WG interface. It's not much, but it's a start.
- UDPATE: traceroute is brilliant for this! For e.g., traceroute 8.8.8.8. (Hat tip to Chrispus Kamau.)
curl ipinfo.io to check your IP address with the VPN enabled. (To be run on the client.)

RELATED:

https://superuser.com/a/1651179