How can I capture IPSEC packets on my VPN server?

Solution 1:

The filter with tcp port 80 will never capture ESP, since esp protocol (IP protocol 50) is not tcp (IP protocol 6) and will never match this filter.

For Linux, this schematic and its few places with xfrm (IPsec & co. transformation module) help to understand how are handled IPsec packets.

On the left side (ingress), a copy of each received packet is sent to AF_PACKET taps (such as tcpdump) and reaches the protocol layer where it's decoded and looped back as plain text, with a new copy sent to AF_PACKET taps: tcpdump will capture two packets for each incoming IPsec packet reaching the interface and then successfully decrypted. So OP's command would still capture the decrypted version of the packet as TCP traffic rather than nothing.

Because ESP can be encapsulated for NAT-T, it can arrive either as pure ESP or as ESP encapsulated (usually) in UDP port 4500.

On the right side (egress), since the packet is already encrypted by xfrm before reaching AF_PACKET, tcpdump can only see the encrypted result and isn't able to see the plain text version. Here, OP's command would have captured nothing. Likewise it can be emitted as pure ESP or encapsulated in 4500/UDP.

So in the end, this should capture the encrypted IPsec traffic both ways:

tcpdump -n -s0 -p -i eth0 -w log.pcap esp or udp port 4500

with a few non-ESP IKE traffic also captured on 4500/UDP, as described there, which can easily be filtered out (SPI field's value is 0) if really needed, even at capture time with this filter instead: 'esp or (udp port 4500 and udp[8:4] != 0)'.

Notes:

• in the rarely used case of transport mode with AH+ESP, the incoming packet would be an AH packet rather than an ESP packet (ah or could be added in front of tcpdump's filter to also address this case)
• on Linux' IPsec topic, sometimes the packet can loop once more in the routing stack when compression is enabled and an uncompressed (usually short) packet is received. There are implications, this time for a firewall, described there. Won't affect tcpdump.

Solution 2:

I was able to capture traffic inside IPSEC using this strongswan page

In short:

iptables -I INPUT -m addrtype --dst-type LOCAL -m policy --pol ipsec --dir in -j NFLOG --nflog-group 5
iptables -I OUTPUT -m policy --pol ipsec --dir out -j NFLOG --nflog-group 5

tcpdump -s 0 -n -i nflog:5 -w ./ipsec.pcap