received netlink error: Invalid argument when trying to connect using ipv6

I'm trying to connect an initiator to a responder using ipv6

Both are on Ubuntu 20.04 and strongswan 5.9.0 Initiator is on a virtual machine (using VMware) on a personal computer behind a box connected to my ISP network. The VM uses a NAT network with the host computer The responder is an online VPS

The initiator got an ipv6 (his host's one) and can ping the responder ipv6. It's ipv6 is <initiator_ipv6> The responder ipv6 seems correctly configured, I can ping and ssh using it. It's ipv6 is <responder_ipv6>

IPv6 have been activated using following conf (as given by my VPS provider)

/etc/netplan/51-ipv6.yaml

network:
    version: 2
    ethernets:
        <responder_interface>:
            dhcp4: true
            dhcp6: false
            match:
              name: <responder_interface>
            addresses:
              - <responder_ipv6>/128
            gateway6: <responder_ipv6_gateway>
            routes:
              - to: <responder_ipv6_gateway>
                scope: link

Both strongswan have been built on the initiator using configure with the following commands :

./configure --prefix=<custom_dir> --with-piddir=<custom_dir>/var/run 
make
make install

They have the same strongswan.conf, which is mostly the default one with a filelog option for easier debug

charon {
        load_modular = yes

        plugins {

                include strongswan.d/charon/*.conf
        }
        filelog {
                logger {
                        path = <custom_dir>/swanctl.log
                        time_format = %b %e %T
                        ike_name = yes
                        default = 1
                        flush_line = yes
                }

        }
}

include strongswan.d/*.conf

The responder has the following swanctl.conf

connections {
       # Seems to change nothing
       local_addrs  = <responder_ipv6> 
       pubkey-vm {
                pools = primary-pool-ipv4, primary-pool-ipv6
                local {
                        auth = pubkey
                        certs = <responder_cert>
                        id = <responder_id>
                }
                remote {
                        auth = pubkey
                        id = <initiator_id>
                }
                children {
                        client {
                        }
                }
        }
}

pools {
    primary-pool-ipv4 {
        # Totally arbitrary value
        addrs = 10.0.0.0/24 
    }
    primary-pool-ipv6 {
        # Totally arbitrary value
        addrs = 2620:0:2d0:200::7/97
    }
}

The initiator has the following conf

connections {
        initiator-to-host {
                remote_addrs = <responder_ip>
                # Any value works, at long as vips got a value (but assigned ip will be in the respodner pool)
                vips = 1.2.3.4
                local {
                        auth = pubkey
                        certs = <initator_cert>
                        id = <initiator_id>
                }

                remote {
                        auth = pubkey
                        id = <responder_id>
                }
                children {
                        to-host {
                                start_action = none
                        }
                }
        }
}

When I try to connect using the responder ipv4, it works and the initiate completed successfully. When I switch the ip to the ipv6 one, it does not work and I got following logs on responder

<pubkey-vm|4> received packet: from <initiator_ipv6>[53870] to <responder_ipv6>[4500] (80 bytes)
<pubkey-vm|4> parsed INFORMATIONAL request 2 [ D ]
<pubkey-vm|4> received DELETE for IKE_SA pubkey-vm[4]
<pubkey-vm|4> deleting IKE_SA pubkey-vm[4] between <responder_ipv6>[<responder_id>]...<initiator_ipv6>[<initiator_id>]
<pubkey-vm|4> IKE_SA deleted
<pubkey-vm|4> generating INFORMATIONAL response 2 [ ]
<pubkey-vm|4> sending packet: from <responder_ipv6>[4500] to <initiator_ipv6>[53870] (80 bytes)
<pubkey-vm|4> lease 10.0.0.1 by '<initiator_id>' went offline
<5> received packet: from <initiator_ipv6>[1016] to <responder_ipv6>[500] (392 bytes)
<5> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
<5> <initiator_ipv6> is initiating an IKE_SA
<5> selected proposal: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_AES128_XCBC/CURVE_25519
<5> remote host is behind NAT
<5> sending cert request for "C=FR, O=Test, CN=Test CA"
<5> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
<5> sending packet: from <responder_ipv6>[500] to <initiator_ipv6>[1016] (273 bytes)
<5> received packet: from <initiator_ipv6>[53870] to <responder_ipv6>[4500] (356 bytes)
<5> parsed IKE_AUTH request 1 [ EF(2/2) ]
<5> received fragment #2 of 2, waiting for complete IKE message
<5> received packet: from <initiator_ipv6>[53870] to <responder_ipv6>[4500] (1220 bytes)
<5> parsed IKE_AUTH request 1 [ EF(1/2) ]
<5> received fragment #1 of 2, reassembled fragmented IKE message (1504 bytes)
<5> parsed IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
<5> received cert request for "C=FR, O=Test, CN=Test CA"
<5> received end entity cert "C=FR, O=Test, CN=<initiator_id>"
<5> looking for peer configs matching <responder_ipv6>[<responder_id>]...<initiator_ipv6>[<initiator_id>]
<pubkey-vm|5> selected peer config 'pubkey-vm'
<pubkey-vm|5>   using trusted ca certificate "C=FR, O=Test, CN=Test CA"
<pubkey-vm|5> checking certificate status of "C=FR, O=Test, CN=<initiator_id>"
<pubkey-vm|5> certificate status is not available
<pubkey-vm|5>   reached self-signed root ca with a path length of 0
<pubkey-vm|5>   using trusted certificate "C=FR, O=Test, CN=<initiator_id>"
<pubkey-vm|5> authentication of '<initiator_id>' with RSA_EMSA_PKCS1_SHA2_256 successful
<pubkey-vm|5> peer supports MOBIKE
<pubkey-vm|5> authentication of '<responder_id>' (myself) with RSA_EMSA_PKCS1_SHA2_256 successful
<pubkey-vm|5> IKE_SA pubkey-vm[5] established between <responder_ipv6>[<responder_id>]...<initiator_ipv6>[<initiator_id>]
<pubkey-vm|5> scheduling rekeying in 14286s
<pubkey-vm|5> maximum IKE_SA lifetime 15726s
<pubkey-vm|5> sending end entity cert "C=FR, O=Test, CN=<responder_id>"
<pubkey-vm|5> peer requested virtual IP 1.2.3.4
<pubkey-vm|5> reassigning offline lease to '<initiator_id>'
<pubkey-vm|5> assigning virtual IP 10.0.0.1 to peer '<initiator_id>'
<pubkey-vm|5> selected proposal: ESP:AES_GCM_16_128
<pubkey-vm|5> received netlink error: Invalid argument (22)
<pubkey-vm|5> unable to add SAD entry with SPI cffdef3f (FAILED)
<pubkey-vm|5> received netlink error: Invalid argument (22)
<pubkey-vm|5> unable to add SAD entry with SPI c028512b (FAILED)
<pubkey-vm|5> unable to install inbound and outbound IPsec SA (SAD) in kernel
<pubkey-vm|5> failed to establish CHILD_SA, keeping IKE_SA
<pubkey-vm|5> generating IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR) N(MOBIKE_SUP) N(ADD_4_ADDR) N(NO_PROP) ]
<pubkey-vm|5> sending packet: from <responder_ipv6>[4500] to <initiator_ipv6>[53870] (1216 bytes)

Client-side logs are here

Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-56-generic, x86_64)
loading ca certificates from '<custom_path>/etc/ipsec.d/cacerts'
loading aa certificates from '<custom_path>/etc/ipsec.d/aacerts'
loading ocsp signer certificates from '<custom_path>/etc/ipsec.d/ocspcerts'
loading attribute certificates from '<custom_path>/etc/ipsec.d/acerts'
loading crls from '<custom_path>/etc/ipsec.d/crls'
loading secrets from '<custom_path>/etc/ipsec.secrets'
loaded plugins: charon aes eap-aka eap-dynamic eap-identity eap-sim eap-tls des rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp curve25519 xcbc cmac hmac drbg attr kernel-netlink resolve socket-default stroke vici updown xauth-generic counters
spawning 16 worker threads
loaded certificate 'C=FR, O=Test, CN=<responder_id>'
loaded certificate 'C=FR, O=Test, CN=<initiator_id>'
loaded certificate 'C=FR, O=Test, CN=Test CA'
loaded RSA private key
loaded RSA private key
added vici connection: host-host
vici initiate CHILD_SA 'to-host'
<host-host|1> initiating IKE_SA host-host[1] to <responder_ipv6>
<host-host|1> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
<host-host|1> sending packet: from <initiator_ipv6>[500] to <responder_ipv6>[500] (392 bytes)
<host-host|1> received packet: from <responder_ipv6>[500] to <initiator_ipv6>[500] (273 bytes)
<host-host|1> parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
<host-host|1> selected proposal: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_AES128_XCBC/CURVE_25519
<host-host|1> local host is behind NAT, sending keep alives
<host-host|1> received cert request for "C=FR, O=Test, CN=Test CA"
<host-host|1> sending cert request for "C=FR, O=Test, CN=Test CA"
<host-host|1> authentication of '<initiator_id>' (myself) with RSA_EMSA_PKCS1_SHA2_256 successful
<host-host|1> sending end entity cert "C=FR, O=Test, CN=<initiator_id>"
<host-host|1> establishing CHILD_SA to-host{1}
<host-host|1> generating IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
<host-host|1> splitting IKE message (1504 bytes) into 2 fragments
<host-host|1> generating IKE_AUTH request 1 [ EF(1/2) ]
<host-host|1> generating IKE_AUTH request 1 [ EF(2/2) ]
<host-host|1> sending packet: from <initiator_ipv6>[4500] to <responder_ipv6>[4500] (1220 bytes)
<host-host|1> sending packet: from <initiator_ipv6>[4500] to <responder_ipv6>[4500] (356 bytes)
<host-host|1> received packet: from <responder_ipv6>[4500] to <initiator_ipv6>[4500] (1216 bytes)
<host-host|1> parsed IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR) N(MOBIKE_SUP) N(ADD_4_ADDR) N(NO_PROP) ]
<host-host|1> received end entity cert "C=FR, O=Test, CN=<responder_id>"
<host-host|1>   using trusted ca certificate "C=FR, O=Test, CN=Test CA"
<host-host|1> checking certificate status of "C=FR, O=Test, CN=<responder_id>"
<host-host|1> certificate status is not available
<host-host|1>   reached self-signed root ca with a path length of 0
<host-host|1>   using trusted certificate "C=FR, O=Test, CN=<responder_id>"
<host-host|1> authentication of '<responder_id>' with RSA_EMSA_PKCS1_SHA2_256 successful
<host-host|1> IKE_SA host-host[1] established between <initiator_ipv6>[<initiator_id>]...<responder_ipv6>[<responder_id>]
<host-host|1> scheduling rekeying in 13877s
<host-host|1> maximum IKE_SA lifetime 15317s
<host-host|1> installing new virtual IP 10.0.0.1
<host-host|1> received NO_PROPOSAL_CHOSEN notify, no CHILD_SA built
<host-host|1> failed to establish CHILD_SA, keeping IKE_SA
<host-host|1> peer supports MOBIKE

Any idea on how to solve that ?

I tried a solution using --enable-kernel-libipsec in configure option. So now I got

./configure --prefix=<custom_dir> --with-piddir=<custom_dir>/var/run --enable-kernel-libipsec 

initiator-side logs remained the same but I got new responder-side logs below

[CFG] <pubkey-vm|1> selected proposal: ESP:AES_GCM_16_128
[ESP] <pubkey-vm|1> failed to create ESP context: unsupported AEAD algorithm AES_GCM_16
[ESP] <pubkey-vm|1> failed to create SAD entry
[ESP] <pubkey-vm|1> failed to create ESP context: unsupported AEAD algorithm AES_GCM_16
[ESP] <pubkey-vm|1> failed to create SAD entry
[IKE] <pubkey-vm|1> unable to install inbound and outbound IPsec SA (SAD) in kernel
[IKE] <pubkey-vm|1> failed to establish CHILD_SA, keeping IKE_SA

Solution 1:

If there is a NAT between your hosts, as detected here by the peers:

<5> remote host is behind NAT

and

<host-host|1> local host is behind NAT, sending keep alives

UDP encapsulation for ESP will be enabled for the IPsec SAs (NAT-Traversal). However, that requires support for it by the IPsec stack, which Linux only provides since 5.8 for IPv6. So with older kernels, this won't work.

You have to either use a newer kernel, remove the NAT, go back to using IPv4, or use an alternative IPsec stack (such as strongSwan's own libipsec).