IPsec site-to-site VPN issues after recent Linux kernel update

Last weekend we had an automatic security upgrade on one of our VPN gateways that connect sites to our cloud environment. After performing troubleshooting (via basic network troubleshooting e.g. via Wireshark) we identified one of the most recent security updates to be the cause of this. We have restored the system back to a known good state and have set (we believe to be) affected packages on hold.

It is a Ubuntu 20.04 LTS instance on AWS with linux-image-aws installed. We are using IPsec to connect several EdgeRouters to a private cloud environment.

After the upgrade all sites connect and communicate as usual, e.g. ICMP is working but we are unable to access certain services (such as RDP or SMB) in the private cloud environment.

The change logs for the related packages don't show any obvious linked change, so I am wondering if I am missing something fundamental. This configuration/setup has worked well for over a year now with no issues.

Known good version: linux-image-aws 5.8.0.1041.43~20.04.13

Problematic version: linux-image-aws 5.8.0.1042.44~20.04.14 and onwards (we have also tested latest 5.11 which seems to be affected)

IPsec configuration extract

# MAIN IPSEC VPN CONFIG
config setup

conn %default
        keyexchange=ikev1

# <REMOVED>
conn peer-rt1.<REMOVED>.net.au-tunnel-1
        left=%any
        right=rt1.<REMOVED>.net.au
        rightid="%any"
        leftsubnet=172.31.0.0/16
        rightsubnet=10.35.0.0/16
        ike=aes128-sha1-modp2048!
        keyexchange=ikev1
        ikelifetime=28800s
        esp=aes128-sha1-modp2048!
        keylife=3600s
        rekeymargin=540s
        type=tunnel
        compress=no
        authby=secret
        auto=route
        keyingtries=%forever
        dpddelay=30s
        dpdtimeout=120s
        dpdaction=restart

Thank you in advance.

EDIT 1: I have been able to capture another tcpdump from an unsuccessful RDP connection after another test upgrade, which looks like the following:

21:43:01.813502 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [S], seq 2706968963, win 64954, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:43:01.813596 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [S], seq 2706968963, win 64954, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:43:01.814238 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [S.], seq 152885333, ack 2706968964, win 64000, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
21:43:01.839105 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1, win 1025, length 0
21:43:01.839168 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1, win 1025, length 0
21:43:01.840486 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 1:48, ack 1, win 1025, length 47
21:43:01.840541 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 1:48, ack 1, win 1025, length 47
21:43:01.843746 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1:20, ack 48, win 63953, length 19
21:43:01.922120 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 20, win 1025, length 0
21:43:01.922212 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 20, win 1025, length 0
21:43:01.932646 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 48:226, ack 20, win 1025, length 178
21:43:01.932729 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 48:226, ack 20, win 1025, length 178
21:43:01.940677 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 20:1217, ack 226, win 63775, length 1197
21:43:01.967343 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 226:408, ack 1217, win 1020, length 182
21:43:01.967417 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 226:408, ack 1217, win 1020, length 182
21:43:01.969452 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1217:1324, ack 408, win 63593, length 107
21:43:02.044376 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1324, win 1020, length 0
21:43:02.044471 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1324, win 1020, length 0
21:43:02.135594 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 408:637, ack 1324, win 1020, length 229
21:43:02.135653 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 408:637, ack 1324, win 1020, length 229
21:43:02.136796 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1324:2609, ack 637, win 63364, length 1285
21:43:02.212871 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 2609, win 1025, length 0
21:43:02.212940 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 2609, win 1025, length 0

Without digging into it - is it perhaps removal of the old, weak ciphers like sha1 and aes128? Try to change it to aes256-sha256-modp2048 in the working kernel version and then upgrade to see if it still breaks. Also maybe ikev2 instead of ikev1? But that’s a userspace concern, not a kernel setting.

Give that a try…

IPsec site-to-site VPN issues after recent Linux kernel update

Related

Recent Posts