openvpn on raspberry pi (pivpn): connects but unusably slow

I set up a raspberry pi 4 2GB as an openvpn server using pivpn. Using my phone to test, I get a connection, but it's too slow to even load a page (< 100 Bit/s). I'm trying to find the error in my setup.

The phone shows it's connected, and when I power off the pi, it does not, so routing and authentication seem to be working:

I'm not sure where to start, so I'll just write down some information that might be relevant:

• The phone's and raspberry pi's connection to the internet are fast enough (for the pi: 33ms ping, 75 MBit/s down, 50 up; for the phone: 17 ms ping, 20 MBit/s down, 12 up). The pi is connected via Wifi, the phone is using its mobile data connection.

• I'm using port forwarding on port 1194, and a ddns service that's updated with ddclient. The LAN uses DHCP with an IP reservation for the pi.

• The setup is using tunnel options V4,dev-type tun,link-mtu 1569,tun-mtu 1500,proto UDPv4,cipher AES-256-CBC,auth SHA256,keysize 256,key-method 2,tls-client and protocol options cipher: AES-256-GCM, digest: NONE, compress: NONE, peer ID: 0

• Using tcpdump (port not 22), there is some traffic before I connect (background communication with the router, some broadcasts), which grows by a lot when I do connect the client. I see whatsapp and facebook show up in some traffic, but it's still mainly the

• The pi is dedicated to this task, with a fresh lite install of raspian. What's on it is needed/installed by ddclient and/or the pivpn script. top shows <1% CPU and <1% RAM usage by openvpn.

• The header of the .ovpn file created on the server, for this client:

client
dev tun
proto udp
remote xxxx.chickenkiller.com 1194
resolv-retry infinite
nobind
remote-cert-tls server
tls-version-min 1.2
verify-x509-name raspberrypivpn_xxxxxxx-24e8-42a0-ac2b-xxxxxxxxx name
cipher AES-256-CBC
auth SHA256
auth-nocache
verb 3

• The file /etc/openvpn/server.conf:

dev tun
proto udp
port 1194
ca /etc/openvpn/easy-rsa/pki/ca.crt
cert /etc/openvpn/easy-rsa/pki/issued/raspberrypivpn_xxxxx.crt
key /etc/openvpn/easy-rsa/pki/private/raspberrypivpn_xxxxx.key
dh none
ecdh-curve prime256v1
topology subnet
server 20.9.0.0 255.255.255.0
# Set your primary domain name server address for clients
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
# Prevent DNS leaks on Windows
push "block-outside-dns"
# Override the Client default gateway by using 0.0.0.0/1 and
# 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of
# overriding but not wiping out the original default gateway.
push "redirect-gateway def1"
client-to-client
client-config-dir /etc/openvpn/ccd
keepalive 15 120
remote-cert-tls client
tls-version-min 1.2
tls-crypt /etc/openvpn/easy-rsa/pki/ta.key
cipher AES-256-CBC
auth SHA256
user openvpn
group openvpn
persist-key
persist-tun
crl-verify /etc/openvpn/crl.pem
status /var/log/openvpn-status.log 20
status-version 3
syslog
verb 3

• The phone is an andriod phone running the openvpn client app

• The openvpn logs on the phone and on the pi do not show any errors.

• I have not been able to test on my laptop or any other device yet.

Is there something obvious I may have done wrong? Let me know if any specifics of my setup are helpful.

Many thanks!

tcpdump before connecting (I've censored/changed/taken out what I thought might need it):

13:23:03.651388 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:23:04.471171 IP 192.168.1.200.41997 > 192.168.1.255.21027: UDP, length 480
13:23:08.771313 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:23:13.686614 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:23:14.178749 ARP, Request who-has 192.168.1.150 tell 192.168.1.101, length 28
13:23:14.178792 ARP, Reply 192.168.1.150 is-at ff:ff:01:37:e4:b1 (oui Unknown), length 28
13:23:14.507295 IP 192.168.1.101.47687 > 192.168.1.255.21027: UDP, length 477
13:23:18.806609 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:23:20.650068 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 70:3a:cb:75:ce:7b (oui Unknown), length 244
13:23:23.721742 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:23:23.722287 ARP, Request who-has 192.168.1.150 (ff:ff:01:37:e4:b1 (oui Unknown)) tell 192.168.1.1, length 46
13:23:23.722342 ARP, Reply 192.168.1.150 is-at ff:ff:01:37:e4:b1 (oui Unknown), length 28

tcpdump in first seconds after connecting:

13:28:11.877479 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 54
13:28:11.878677 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 66
13:28:11.992044 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 339
13:28:11.992858 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 62
13:28:11.993104 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 153
13:28:12.004387 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 378
13:28:12.013758 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 1128
13:28:12.014035 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 503
13:28:12.019489 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 62
13:28:12.022810 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 1316
13:28:12.026282 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 355
13:28:12.033375 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 224
13:28:12.034349 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 287
13:28:12.038947 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 62
13:28:12.040022 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 62
13:28:12.041430 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 89
13:28:12.042633 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 62
13:28:12.042836 IP 192.168.1.150.openvpn > 192.168.1.1.43448: UDP, length 300
13:28:12.078304 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 62
13:28:12.079115 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 140
13:28:12.079177 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 72
13:28:12.079214 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 140
13:28:12.161953 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:12.163013 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.167330 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:12.167365 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.171647 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 100
13:28:12.171840 IP 20.9.0.3.54810 > dns.google.domain: 37821+ A? infinitedata-pa.googleapis.com. (48)
13:28:12.172282 IP 192.168.1.150.52627 > 192.168.1.1.domain: 36332+ PTR? 3.0.8.10.in-addr.arpa. (39)
13:28:12.174861 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 102
13:28:12.175040 IP 20.9.0.3.16393 > dns.google.domain: 59285+ A? digitalassetlinks.googleapis.com. (50)
13:28:12.175944 IP 192.168.1.1.domain > 192.168.1.150.52627: 36332 NXDomain* 0/0/0 (39)
13:28:12.176222 IP 192.168.1.150.47687 > 192.168.1.1.domain: 2044+ PTR? 8.8.8.8.in-addr.arpa. (38)
13:28:12.191657 IP 192.168.1.1.domain > 192.168.1.150.47687: 2044 1/0/0 PTR dns.google. (62)
13:28:12.282279 IP 192.168.1.22.mdns > 224.0.0.251.mdns: 1 [2q] PTR (QU)? _CC32E753._sub._googlecast._tcp.local. PTR (QU)? _googlecast._tcp.local. (61)
13:28:12.358935 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 85
13:28:12.359554 IP 20.9.0.3.37234 > dns.google.domain: 61029+ NAPTR? rbm.mavenir.com. (33)
13:28:12.386618 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.390037 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.508233 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 84
13:28:12.508746 IP 20.9.0.3.49488 > 57.74.98.34.bc.googleusercontent.com.https: Flags [S], seq 72412679, win 65535, options [mss 1361,sackOK,TS val 2637578 ecr 0,nop,wscale 8], length 0
13:28:12.509118 IP 192.168.1.150.42940 > 192.168.1.1.domain: 11656+ PTR? 57.74.98.34.in-addr.arpa. (42)
13:28:12.533377 IP 192.168.1.1.domain > 192.168.1.150.42940: 11656 1/0/0 PTR 57.74.98.34.bc.googleusercontent.com. (92)
13:28:12.626709 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.630095 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:12.660407 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:12.761745 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 140
13:28:12.792873 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 84
13:28:12.793170 IP 20.9.0.3.4176 > dns.google.domain: 36977+ A? g.whatsapp.net. (32)
13:28:12.873592 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 140
13:28:12.894501 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:13.011053 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 88
13:28:13.011355 IP 20.9.0.3.26958 > dns.google.domain: 28809+ AAAA? dealer.spotify.com. (36)
13:28:13.012418 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 88
13:28:13.012701 IP 20.9.0.3.53255 > dns.google.domain: 34834+ A? dealer.spotify.com. (36)
13:28:13.088709 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:13.092741 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:13.134467 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:13.176740 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:13.177032 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:13.309546 IP 192.168.1.22.mdns > 224.0.0.251.mdns: 2 [2q] PTR (QM)? _CC32E753._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (61)
13:28:13.521911 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 84
13:28:13.522240 IP 20.9.0.3.49488 > 57.74.98.34.bc.googleusercontent.com.https: Flags [S], seq 72412679, win 65535, options [mss 1361,sackOK,TS val 2637880 ecr 0,nop,wscale 8], length 0
13:28:13.594254 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:13.972678 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:13.975944 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 211
13:28:14.128750 ff:ff:37:ef:1b:c4 (oui Unknown) > Broadcast, ethertype Unknown (0x6970), length 74: 
    0x0000:  010e 58dd dddd b8e9 37ef 1bc4 6970 0101  ..X.....7...ip..
    0x0020:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5 a5a5  ................
    0x0030:  a5a5 a5a5 a5a5 a5a5 a5a5 a5a5            ............
13:28:14.315567 IP 192.168.1.22.mdns > 224.0.0.251.mdns: 3 [2q] PTR (QM)? _CC32E753._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (61)
13:28:14.321486 IP 192.168.1.101.47687 > 192.168.1.255.21027: UDP, length 477
13:28:14.480595 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:14.973356 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 84
13:28:14.973631 IP 20.9.0.3.49178 > 25.224.186.35.bc.googleusercontent.com.https: Flags [S], seq 1821028633, win 65535, options [mss 1361,sackOK,TS val 2638316 ecr 0,nop,wscale 8], length 0
13:28:14.974283 IP 192.168.1.150.41045 > 192.168.1.1.domain: 13203+ PTR? 25.224.186.35.in-addr.arpa. (44)
13:28:14.993607 IP 192.168.1.1.domain > 192.168.1.150.41045: 13203 1/0/0 PTR 25.224.186.35.bc.googleusercontent.com. (96)
13:28:15.267765 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:15.268366 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 76
13:28:15.409468 IP 192.168.1.1.43448 > 192.168.1.150.openvpn: UDP, length 93
13:28:15.409698 IP 20.9.0.3.24907 > dns.google.domain: 25615+ A? spclient.wg.spotify.com. (41)

With much help from Tom, the problem turned out to be in the iptables configuration.

sudo iptables-save returns:

# Generated by xtables-save v1.8.2 on Sun Jun 20 15:47:29 2021
*filter
:INPUT ACCEPT [1308:157588]
:FORWARD ACCEPT [483:31962]
:OUTPUT ACCEPT [431:1989618]
COMMIT
# Completed on Sun Jun 20 15:47:29 2021
# Generated by xtables-save v1.8.2 on Sun Jun 20 15:47:29 2021
*nat
:PREROUTING ACCEPT [291:34883]
:INPUT ACCEPT [50:18696]
:POSTROUTING ACCEPT [252:17084]
:OUTPUT ACCEPT [11:897]
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -m comment --comment openvpn-nat-rule -j MASQUERADE
COMMIT

Note the -o eth0. I am actually using the wireless interface.

Removing it from /etc/iptables/rules.v4 and rebooting, the vpn connection is now working as expected. I can reach IP addresses in my LAN, and the connection is fast:

Some remarks:

• The openvpn client on android still shows low speeds, even while e.g. watching youtube. It threw me off a bit, but I guess this is just the overhead for the vpn tunnel itself, encryption.

• I had not manually entered the -o eth0 option to the iptables - I assume it was added by the pivpn script. I did briefly try to use the ethernet port, but not while most recently setting up/reconfiguring pivpn. Not sure if this is a bug, but at least it's something to keep in mind when setting up a vpn server.

I hope this is helpful to others who stumble upon this question. Many thanks again to Tom