OpenVPN: How to mitigate path MTU issues on a per-client basis?

Solution 1:

I solved the problem on the client side by adding the option mssfix 1300 to the config file.

From the openvpn man page:

--mssfix max
    Announce to TCP sessions running over the tunnel that they should limit their send packet sizes such that after OpenVPN has encapsulated them, the resulting UDP packet size that OpenVPN sends to its peer will not exceed max bytes. 

Original idea for my solution came from personalvpn.org

Solution 2:

Given the lack of answers, i am posting now a solution that is not very elegant, but simple: Run another OpenVPN instance on TCP for "bad clients"

proto tcp

and lower the TCP MSS on the client, e.g.

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o ${OUT_DEV} -j TCPMSS --set-mss ${PATH-MTU-MINUS-40}

An advantage of this solution is that each client can set its individual MSS.

This is admittedly TCP-over-TCP, but that should perform well enough in many scenarios.

Note that I am still very interested solutions that don't require proto tcp, and i'll mark them as valid answer if they more or less fulfil my outlined requirements.

Solution 3:

Not sure if this can help with embedded devices, but I'm sharing my experience in case it can help others.

TL;DR: skip to the last question/answer for a possible solution

• Why exactly are link-mtu and tun-mtu discouraged? What are the potential problems with these options? Note that i am quite comfortable with low-level IP header munging.

In my experience as user of OpenVPN, for most common cases it is actually the fragment and/or mssfix values that are better not be fiddled with, i.e. contrary to what the documentation states.

link-mtu and tun-mtu values are computed related to each other depending on which cipher (if any) is used for the tunnel. By default tun-mtu is meant to be the classic 1500 bytes while link-mtu is derived on that, but you may set either one and thus have the other recomputed accordingly. In this regard, one quite-certain rule of thumb is to never set both: only set either one, because the other will just follow suit.

The only case where I did run into problems while altering them is when I set the endpoints to have very different values and with the server being the one having a very low value, like 576 on it while 1500 on the client. These set-ups simply do not handshake at all, and I have never investigated further because, frankly, I've only had such set-ups in artificial lab-tests only. The opposite case, i.e. very low value on client while normal/high value on server, works just fine.

In fact, I have found that setting either link-mtu or tun-mtu (I mostly prefer setting link-mtu) actually solves most common cases just neatly, i.e. the tunnel works just as it should including any non-TCP traffic.

Of course, setting fragment and/or mssfix values (if those are also explicitly set) must take the explicit link-mtu/tun-mtu values into account, but I found that just leaving them alone (i.e. not even mentioned in the configuration) is usually the best thing to do because they simply adjust automatically according to the link-mtu/tun-mtu values.

That said, it is true that setting only mssfix (thus leaving link-mtu/tun-mtu as well as fragment unspecified) does solve everything related to TCP, but all other protocols are bound to experience problems, the only reason why they usually don't is simply because most non-TCP traffic is either DNS on UDP, which is typically low sized, or ICMP echo-requests/replies, which are again small packets by default.

Note however that I'm referring to OpenVPN on Linux; on other OSes, particularly the Windows series and iOS devices, it might be a different story, i.e. altering link-mtu/tun-mtu might be irrelevant or even disruptive and you might have to resort to altering fragment/mssfix as suggested by the documentation. Particularly hostile cases may be when either the server or the clients (or both) do not support IP fragmentation/reassembly that may be triggered by lowering link-mtu/tun-mtu artificially, and thus in such cases you can only resort to enable fragment in OpenVPN.

• Which of the options link-mtu tun-mtu fragment mssfix have to be mirrored on the server-side in order to work?

According to the warning messages issued by OpenVPN to its log when you set either link-mtu/tun-mtu while not mirroring them to the endpoint, the link-mtu and tun-mtu values should be exactly mirrored, either explicitly or implicitly. However, in real use-cases I have never experienced problems in altering either value even when I ended up with very different values between the tunnel endpoints.

fragment must either be present on both sides or absent on both sides, because its presence enables fragmentation performed by OpenVPN itself using its own internal algorithm (not IP fragmentation), and as such it must be agreed upon by both endpoints. However, when it is present, it may have asymmetric values.

mssfix does not need to be mirrored, and it may also have asymmetric values between the endpoints.

• Which of the options link-mtu tun-mtu fragment mssfix can be used in client-config-dir?

None

A possible solution

• In case all four options have to be mirrored server-side, and cannot be used inside client-config-dir: Are there any alternatives to combat low path MTU per client?

One way is by using Linux's per-route settings, which allows setting MTU values arbitrarily, up to the MTU of the actual interface. It goes as follows:

For starters, perform Path MTU Discovery manually (e.g. using ping -M do -s xxxx <client's-*public*-address> commands from the server towards the remote clients) in order to find out the correct MTU value between the server and each specific remote client. Then,

on the client side, assuming it is just a client and not a router for other hosts in its LAN, just set OpenVPN's link-mtu value to the actual MTU minus 28. For instance, on a remote client having an actual MTU of 576 an link-mtu 548 will do the trick.

Note that it will do for all traffic (not just TCP) originating from the client, because the link-mtu value is used by OpenVPN as an upper-bound size for its own (UDP port 1194) payloads sent to the remote endpoint, hence the remote client's actual MTU (as discovered manually) minus 20 (outer IP header size without options) minus 8 (outer UDP header size).

The MSS value that OpenVPN will clamp automatically to in-tunnel TCP traffic will then be link-mtu minus OpenVPN's own overhead (which varies depending on cipher used), minus 20 (inner IP header size without options) minus 20 (inner TCP header size without options).

on the server side, install one route per each "low-MTU" client, in order to set the correct MTU for each of them. Each correct value to set on the per-route settings has to be what you set on that remote client's link-mtu value (as determined previously) minus OpenVPN's own overhead. This latter can be derived from the link-mtu value minus the tun-mtu value reported by the OpenVPN server for that specific tunnel. For instance, assuming the OpenVPN server reports a link-mtu of 1541 and a tun-mtu of 1500, then on the machine hosting your OpenVPN server you would do something like:

ip route add <client's-*vpn*-address> via <your-openvpn-server's-*vpn*-address> mtu 507

Such operation can be done conveniently on-demand by a client-connect script, which receives those values (as well as many others) in environment variables set dynamically by OpenVPN. In shell-script parlance it would be:

ip route replace "$ifconfig_pool_remote_ip" via "$ifconfig_local" mtu "$(( 576 - 28 - (link_mtu - tun_mtu) ))"

From then on, outbound traffic towards that specific client's VPN address will respect that MTU instead of the tun interface's one.

Furthemore, if your OpenVPN server also acts as router (sysctl net.ipv4.ip_forward=1) between its LAN and the remote clients, then the per-route settings applied to the OpenVPN server machine (its Linux kernel) will also trigger proper ICMP messages (type 3 code 4, Fragmentation Needed) towards the machines in LAN when these send DF traffic towards the remote clients, in which case these other machines in LAN must comply to those ICMP messages, and it will also perform IP fragmentation inside the tunnel on behalf of the machines in LAN when these send non-DF traffic, in which case your remote clients must support IP reassembly for the IP fragments coming out of the tunnel.

Note also that on Ubuntu 14.04 HWE and newer (or equivalent kernels up to v5.7.x) you will also have to set sysctl net.ipv4.ip_forward_use_pmtu=1 in order to have Linux perform said ICMP/fragmentation when forwarding other machines's traffic, whereas that additional sysctl is not required for outbound traffic originated by the server machine itself.

Finally note that for a fully correct configuration you should also set link-mtu 1472 (assuming an underlying interface's MTU of 1500) on the server side. I actually do it any time, any where, as a base configuration, except for peculiar cases requiring specific workarounds. This is because OpenVPN does not consider the underlying interface's MTU as a starting value for its link-mtu/tun-mtu, nor does it perform PMTU discovery even when its mtu-disc option is set to yes on OSes that support it. Therefore, explicitly setting link-mtu to a value reflecting the underlying interface's MTU (using the formulae I described above) is to me the true best serious default at least on Linux.

HTH