strongSwan setup where both sides are behind NAT

Solution 1:

Due to the certificates and certificate requests IKE_AUTH messages can get quite large, so much so that they have to be fragmented on the IP layer (you can see those fragments in the tcpdump capture at venus). Perhaps the NAT box at sun has problems reassembling fragmented packets or just drops them.

As a workaround, you can try installing the two peer's certificates on both sides, then configure rightcert accordingly so that it points to the file containing the certificate of the other peer.

With that done, you can configure rightsendcert=never on both ends, to avoid that certificate requests are being sent. Because leftsendcert defaults to ifasked the peers ultimately won't send their certificates and the message size should be small enough to avoid IP fragments.

By the way, you don't have to open UDP port 50. Without NAT traversal you'd need to allow IP protocol 50 (ESP), but if a NAT is involved ESP packets get UDP encapsulated so opening UDP ports 500 and 4500 is sufficient.