What ports need to be opened to use the L2TP VPN server on Mountain Lion Server?

I am using a Mac mini OS X Mountain Lion Server (10.8.5). My goal is to have the server's own VPN service running over L2TP only.

The Mac mini is behind an AirPort Extreme (4th generation)

AirPort Extreme has:

  • static IPv4 address

  • no connection sharing (bridged mode)

  • Back To My Mac disabled (as for all the Macs and AirPorts on the network)

  • Internet connection via Ethernet to my ISP's box

ISP's box:

  • has static public IP address

  • is the DHCP server with

    • IP addresses reservations configured for my AirPort Extreme & Mac mini and

    • private IP addresses from x.x.x.10 up-to x.x.x.50

  • has NAT/PAT rules for

    • UDP 500 - ISAKMP/IKE

    • UDP 4500 - IPsec NAT-Traversal

    • UDP 1701 - L2TP

    • ESP/IP 50

    • AH/IP 51

Mac mini Server:

  • has static IP address

  • is the DNS server for the network

  • is (of course) the VPN server with the configuration as follows:

    • setup for: L2TP only

    • VPN hostname: public IP address

    • Shared secret: dull-8caracter word

    • Addresses: 10 for L2TP x.x.x.200 up-to x.x.x.209

    • Name server:
      my MacMini private-static IP address + ISP'Box private IP address + google DNS

    • Routes:
      x.x.x.0 255.255.255.0 (my private network) private
      0.0.0.0 0.0.0.0 public

NB: My ISP is NOT blocking any ports.

I am aware of the "advanced" guide provided by Apple.

I have been able to access the server when inside my network (-_-)

Trying to connect from 3G network with iPhone 4 gives

"L2TP VPN server did not respond"

Trying to connect from 3G OR other ADSL network with Windows8 gives

"error 789 the L2TP connection attempt failed because the security layer encountered a processing error during initial negociations with the remote computer"

Is this real life?


Solution 1:

According to Apple, L2TP is currently inoperative when the VPN server is NATed. Looks like we have to wait for a bug fix, or use PPTP in the mean time despite it's much lower security.

Remember, if you configure PPTP, you need to activate OpenDirectory, and configure users there. Local users CANNOT use PPTP.

References: http://support.apple.com/kb/TS5313 http://support.apple.com/kb/HT4748