FreeBSD Jail with Loopback IP, IPFW, and natd - Outbound connections fail from jail

I've got a FreeBSD 9.0 server. It has several jails, but they all have this same issue. They can not initiate connections to the outside world. They communicate with each other and the host fine.

relevant rc.conf settings:

firewall_enable="YES"                   # IPFW
firewall_type="/etc/ipfw.rules"         # Rule script for IPFW

natd_enable="YES"                       # NAT for Internet Routing
natd_interface="wan0"                   # NAT Card
natd_flags="-f /etc/natd.conf -dynamic" # NAT Conf

ifconfig_lo1_name="jail1"
ifconfig_jail1="inet 192.168.1.101/32"

jail_asdf_rootdir="/jails/asdf"
jail_asdf_hostname="asdf.example.net"
jail_asdf_ip="192.168.1.101"
jail_asdf_devfs_enable="YES"

From sysctl.conf

security.jail.allow_raw_sockets=1

From ipfw.rules

# XXX 00050 divert natd ip4 from any to any via wan0
add 00060 check-state

# Allow me out
add 00135 allow ip from me to any keep-state
add 00136 allow ip6 from me6 to any keep-state

# HTTP
add 11010 allow tcp from any to me http setup keep-state
add 11011 allow tcp from any to me6 http setup keep-state
add 11012 allow tcp from any to me https setup keep-state
add 11013 allow tcp from any to me6 https setup keep-state
.... lots more rules like the above ....

# General Network - ICMP
add 61001 allow icmp from any to any

# XXX last rule is deny everything

From natd.conf

redirect_port tcp 192.168.1.101:80 80
redirect_port tcp 192.168.1.101:443 443

This works great for incoming connections. The I have tested from multiple computers, I can get to the website just fine.

When I jexec 1 csh to get a shell in the Jail I can't create an outgoing connection. The Jail's resolv.conf points to the host server, and it performs name resolution fine. With ICMP still being passed without exception I can ping from the Jail.

I can do a tcpdump -i wan0 host 1.2.3.4 on the host and watch the traffic as it passes. I see the SYN go out, and a SYN ACK come back. Then a few seconds later the same again as the Jail retries.

How can I allow outgoing connections from my jails?

Update:
I believe I understand the problem. The outgoing packet starts through the firewall rules, gets NAT translated, allowed out and recorded as the external IP making an outgoing connection. When the return packet comes back it goes through translation, but now doesn't match the check-state rule as the packet has the internal IP. Still searching for a solution.


The solution should have been obvious from the problem that the address translation was always occurring before state rules were checked. The address translation needs to be split.

The corrected version of the rules found above is:

add 00050 divert natd ip4 from any to any via wan0 in
add 00060 check-state

# Talking to myself
add 00200 allow ip from me to me keep-state

# HTTP
add 11010 skipto 63000 tcp from any to me http,https setup keep-state
add 11011 skipto 63000 tcp from any to me6 http,https setup keep-state

# General Network - ICMP
add 61001 allow icmp from any to any

# Last rule of "normal" traffic
add 62000 deny ip from any to any

# Only for my outbound and specifically allowed incoming
add 63000 divert natd ip from any to any via wan0 out
add 63001 allow ip from any to any

# XXX last rule is deny everything