Docker suddenly stopped working with Wireguard, unable to fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz during build

Wireguard and Docker have been working for quite a while together under this same configuration, but this morning I started working and I couldn't get my docker containers to talk to AWS properly.

I have a feeling this is some kind of network change at the co-working space I'm at that's conflicting with my machine's configuration. The only way I've been able to confirm that is if I turn WG off, docker build + running the containers starts working again, or if I keep WG on but connect to my phone's hotspot, it also works.

Furthermore, I couldn't build containers via docker-compose. The simplest way I've been able to replicate the problem is via:

jacob@jacob-ThinkPad-P52s:~$ docker run --rm -ti alpine apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz

Running that container with apk update just ends in an endless hang trying to get APKINDEX.tar.gz. If I navigate to the same URL on the host OS, I don't have any issues.

I have a similar issue trying to contact the AWS SES service from within the docker containers, this just hangs forever:

jacob@jacob-ThinkPad-P52s:~$ docker run --rm -ti alpine wget https://email.us-east-1.amazonaws.com
Connecting to email.us-east-1.amazonaws.com (54.157.160.123:443)

Doing the same from the host machine gives a response immediately. The container is running on a non-default 10.254.0.0/16 subnet (has an IP of 10.254.0.1).

The co-working space is using the 10.0.0.0/16 subnet:

3: wlp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:3b:8f:32:92:2c brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.24/16 brd 10.1.255.255 scope global dynamic noprefixroute wlp4s0
       valid_lft 80541sec preferred_lft 80541sec
    inet6 fe80::c263:54cf:5e8c:a865/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Wireguard configuration:

jacob@jacob-ThinkPad-P52s:~$ sudo wg showconf wg0
[Interface]
ListenPort = 51222
FwMark = 0xca6c
PrivateKey = <private key>
DNS = 172.31.0.2

[Peer]
PublicKey = <public key>
AllowedIPs = 0.0.0.0/0
Endpoint = <server ip>:41194
PersistentKeepalive = 15

Some other potentially useful info:

jacob@jacob-ThinkPad-P52s:~$ ip route
default via 10.1.0.1 dev wlp4s0 proto dhcp metric 600 
10.1.0.0/16 dev wlp4s0 proto kernel scope link src 10.1.1.24 metric 600 
10.254.0.0/16 dev docker0 proto kernel scope link src 10.254.0.5 
10.255.100.0/24 dev br-b06c6d86e92b proto kernel scope link src 10.255.100.1 
169.254.0.0/16 dev wlp4s0 scope link metric 1000 
172.30.0.0/21 dev wg0 proto kernel scope link src 172.30.0.2

jacob@jacob-ThinkPad-P52s:~$ uname -a
Linux jacob-ThinkPad-P52s 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

jacob@jacob-ThinkPad-P52s:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

I'm honestly pretty past my abilities currently and am kind of firing in the dark here. Any help troubleshooting would be immensely appreciated.

Coworking space recently switched from Charter to AT&T and something about the MTU being 1500 on Docker was too large I guess. Seems like AT&T drops fragmented packets which was resulting in infinite hangs. Setting the MTU to a lower value, 1420, instantly solved the issue.