iptables rules to allow L2TP/IPSEC VPN behind firewall

I asked about this on the Smoothwall Express forums but they are understandably reluctant to provide to much help. After all, they really want to sell the commercial version.

I need to provide an L2TP/IPSEC VPN for remote support of some new machinery we're getting soon (no choice about that part). As the remote user also needs to be authenticated against Active Directory I need to run the VPN on our Windows (2003) server, rather than directly on the firewall.

Because Smoothwall Express has no provision for forwarding protocols from it's normal management interface this needs to be done by editing the iptables settings in rc.firewall.up. Herein lies my problem. I don't know iptables and feel that even if I take the time to read up about it the risks of me screwing it up are significant.

The following section currently exists in rc.firewall.up:

# IPSEC
/sbin/iptables -N secin
/sbin/iptables -A secin -i ipsec0 -j ACCEPT
/sbin/iptables -A INPUT -j secin

/sbin/iptables -N secout
/sbin/iptables -A secout -i ipsec0 -j ACCEPT
/sbin/iptables -A FORWARD -j secout

[some unrelated stuff here]

# IPSEC
/sbin/iptables -N ipsec
/sbin/iptables -A ipsec -p udp --destination-port 500 -j ACCEPT
/sbin/iptables -A ipsec -p udp --destination-port 4500 -j ACCEPT
/sbin/iptables -A ipsec -p 50 -j ACCEPT
/sbin/iptables -A ipsec -p 51 -j ACCEPT
/sbin/iptables -A block -i ppp0 -j ipsec
/sbin/iptables -A block -i ippp0 -j ipsec
if [ "$RED_DEV" != "" ]; then
    /sbin/iptables -A block -i $RED_DEV -j ipsec
fi

Unless I'm mistaken that second bock will cause the traffic I want forwarded to be swallowed instead. I tried commenting out that second block and adding this one, which is adapted from the lines required for PPTP passthrough (which does work) and what I've been able to obtain from various sources:

# L2TP/IPSEC
/sbin/iptables -N l2tp
/sbin/iptables -A l2tp -p udp --destination-port 500 --dst 192.169.0.7 -j ACCEPT
/sbin/iptables -A l2tp -p udp --destination-port 1701 --dst 192.169.0.7 -j ACCEPT
/sbin/iptables -A l2tp -p udp --destination-port 4500 --dst 192.169.0.7 -j ACCEPT
/sbin/iptables -A l2tp -p 50 --dst 192.169.0.7 -j ACCEPT
/sbin/iptables -A l2tp -p 51 --dst 192.169.0.7 -j ACCEPT
/sbin/iptables -I FORWARD -j l2tp
/sbin/iptables -t nat -N l2tp
/sbin/iptables -t nat -A l2tp -i $RED_DEV -p tcp --dport 500 -j DNAT --to 192.169.0.7:500
/sbin/iptables -t nat -A l2tp -i $RED_DEV -p tcp --dport 1701 -j DNAT --to 192.169.0.7:1701
/sbin/iptables -t nat -A l2tp -i $RED_DEV -p tcp --dport 4500 -j DNAT --to 192.169.0.7:4500
/sbin/iptables -t nat -A l2tp -i $RED_DEV -p 50 -j DNAT --to 192.169.0.7
/sbin/iptables -t nat -A l2tp -i $RED_DEV -p 51 -j DNAT --to 192.169.0.7
/sbin/iptables -t nat -A PREROUTING -j l2tp

.. but that doesn't work. Where did I stuff up?

Incidentally, $RED_DEV translates to the public facing interface and obviously 192.168.0.7 is my VPN server.

Update:

It turns out that the above settings do work - sort of. I'm testing this from home, where I have a MacBook and a several of Windows XP machines. I can connect just fine from the Mac but cannot connect using any of the Windows machines. Unfortunately, the machinery company that requires this VPN for remote support uses only Windows XP. :(

Solution 1:

As noted in the update to the question, even with double NAT the VPN is working from a Mac to Windows but not Windows to Windows. It seems it's really only a matter of coming up with the correct search term to find a solution.

As per this KB article, on the Windows XP client machine we need to create a new DWORD value named AssumeUDPEncapsulationContextOnSendRule in HKLM\System\CurrentControlSet\Services\IPSec. For double NAT it needs a value of 2.

Now I need to find a solution for Windows 7, because it's inevitable that sooner or later I'm going to have to deal with that.

Update

For the benefit of anyone with the same issue, the fix for Windows 7 is to create a new DWORD value named AssumeUDPEncapsulationContextOnSendRule in HKLM\SYSTEM\CurrentControlSet\services\PolicyAgent. Again, for double NAT it needs a value of 2.