Wireshark and IPSec

I am trying to troubleshoot communication between two servers on a Windows network where IPSEC is encrypting everything. I installed wireshark on the source server and captured traffic at the point that the communication is failing, but other than a few ARPs and DNS packets, everything else that is captured is an ESP (Encapsultating Security Payload) encrypted packet.

I would understand this if I were doing a man-in-the-middle capture, but I am on the source machine. Is there a way to specify that Wireshark capture farther up the stack (after the decryption is complete)? Source machine is W2K8R2 running as a Hyper-V VM if it matters.

If you want to inspect and analyse ESP traffic directly your version of Wireshark needs to be linked with libcrypt. More details here.

To answer my own question (or at least to mention my solution), Netmon is able to capture and parse the same traffic with no problems. I saved the Netmon capture and opened it in Wireshark, and everything still shows up as ESP packets. Apparently Wireshark doesn't like unencrypting the packets. Maybe Netmon uses the local key to do so? In any case, the answer was to use Netmon. It is not nearly as good for analyzing traffic, but it does open ESP packets if you capture them from an endpoint.

You probably just need to tell Wireshark to capture on the virtual interface provided by the IPSec VPN service, rather than on the actual interface. Go to capture->interfaces or to capture->options and select the interface from the dropdown.

In Wireshark, go to Edit/Preferences and expand the Protocol list. Find ESP in the list and enter your key information.