UFW Allows 22 for IPv4 and IPv6 but SSH Disconnects When Enabling
sudo ufw disable followed by sudo ufw enable kicks me out of SSH
DMESG reports
[UFW BLOCK] IN=eth0 OUT= MAC=30:........ SRC=192.168.1.me DST=192.168.1.server LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=15776 DF PROTO=TCP SPT=55640 DPT=22 WINDOW=253 RES=0x00 ACK URGP=0
I can log back in without having to change rules via the console (UFW still enabled).
This started after upgrading Xenial (16.04) from kernel 4.4 to 4.15 (HWE). Upgrading to 18.04.1 did not solve the issue.
Versions:
- iptables v1.6.1
- ufw 0.35
- 4.15.0-29-generic #31-Ubuntu
- Ubuntu 18.04.1 LTS
UFW status verbose is (some rules were omitted, but they are all ALLOW)
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22 ALLOW IN Anywhere
22 (v6) ALLOW IN Anywhere (v6)
Why is this happening, or at least, how to revert to the expected behavior?
I looked at this answer, and I am not sure it applies, but here's /etc/ufw/before.rules
#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
# ufw-before-input
# ufw-before-output
# ufw-before-forward
#
# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines
# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT
# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type source-quench -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type source-quench -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT
# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT
#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local
# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT
PS: I didn't expect this to "fix" the issue but just for reference I changed the port SSHD listens on (and the corresponding rule) and the problem persists.
Solution 1:
Background, and bounds for the issue:
- The issue only occurs when UFW, or iptables, with these ssh allow rules, is enabled and an ssh session is started. i.e. Any SSH session that was started with no iptables at all works fine, but may be subject to random drop outs once the rule set are put in place.
- recall that ufw is merely a front end for iptables.
- The issue is present even with kernel 4.18-rc8.
What is going on?
The sudo ufw allow in port 22 results in the following iptables rules segment:
Chain ufw-before-input (1 references)
pkts bytes target prot opt in out source destination
16 1553 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
386 300622 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
15 1068 ufw-logging-deny all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
15 1068 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
Upon sudo ufw disable followed by sudo ufw enable, and even though the ssh connection itself remains fine, the resulting iptables rule set seems to have forgotten the association with that particular connection and therefore classifies any incoming packets as invalid. Somehow the connection tracking table has become confused and the packet is not even considered NEW, but with incorrect flags, nor is it considered part of the existing connection.
Consider a very basic iptables equivalent of what ufw is doing. Two scripts, one for clearing the rule set and one for creating it:
#!/bin/sh
FWVER=0.01
#
# clear_firewall_min 2018.08.10 Ver:0.01
# clear iptables minimum.
# Currently for this question:
# https://askubuntu.com/questions/1059781/ufw-allows-22-for-ipv4-and-ipv6-but-ssh-disconnects-when-enabling
#
echo "Loading clear_firewall_min version $FWVER..\n"
# The location of the iptables program
#
IPTABLES=/sbin/iptables
#Set some stuff
#
EXTIF="ens5"
UNIVERSE="0.0.0.0/0"
#Clearing any previous configuration
#
echo " Clearing any existing rules and setting default policies.."
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
# Reset all IPTABLES counters
$IPTABLES -Z
#sleep 10
echo clear_firewall_min $FWVER done.
And:
#!/bin/sh
#
# test_firewall 2018.08.13 Ver:0.01
# Minimum version of most basic iptables firewall.
#
# test_firewall 2018.08.09 Ver:0.01
# Most basic iptables firewall.
# Currently for this question:
# https://askubuntu.com/questions/1059781/ufw-allows-22-for-ipv4-and-ipv6-but-ssh-disconnects-when-enabling
#
#sleep 50
# The location of the iptables program
#
IPTABLES=/sbin/iptables
#Set some stuff
#
EXTIF="ens5"
UNIVERSE="0.0.0.0/0"
#Clearing any previous configuration
#
#echo " Clearing any existing rules and setting default policies.."
$IPTABLES -P INPUT DROP
$IPTABLES -F INPUT
# loopback interfaces are valid.
#
$IPTABLES -A INPUT -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate INVALID -j LOG --log-prefix "IINVALID:" --log-level info
$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate INVALID -j DROP
$IPTABLES -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j LOG --log-prefix "NEW TCP no SYN:" --log-level info
$IPTABLES -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j DROP
$IPTABLES -A INPUT -i $EXTIF -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate NEW --dport 22 -j ACCEPT
echo "test_firewall_min $FWVER done..." >> /dev/kmsg
sleep 3
Resulting in these packets counts after a clear/load cycle with an ssh session that was started after a load cycle:
doug@s17:~$ sudo iptables -v -x -n -L
Chain INPUT (policy DROP 3 packets, 220 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
35 6388 LOG tcp -- ens5 * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID LOG flags 0 level 6 prefix "IINVALID:"
35 6388 DROP tcp -- ens5 * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 ctstate NEW LOG flags 0 level 6 prefix "NEW TCP no SYN:"
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 ctstate NEW
9 680 ACCEPT all -- ens5 * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT tcp -- ens5 * 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:22
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 12 packets, 1408 bytes)
pkts bytes target prot opt in out source destination
Notice the 35 invalid packets as I typed on the crippled ssh session terminal, and before PuTTY terminated.
Why did this stop working, it used to work?
Because this is 100% repeatable, a kernel bisection was relatively easy, just time consuming. The results were:
4d3a57f23dec59f0a2362e63540b2d01b37afe0a is the first bad commit
commit 4d3a57f23dec59f0a2362e63540b2d01b37afe0a
Author: Florian Westphal <[email protected]>
Date: Fri Jul 28 11:22:04 2017 +0200
netfilter: conntrack: do not enable connection tracking unless needed
Discussion during NFWS 2017 in Faro has shown that the current
conntrack behaviour is unreasonable.
Even if conntrack module is loaded on behalf of a single net namespace,
its turned on for all namespaces, which is expensive. Commit
481fa373476 ("netfilter: conntrack: add nf_conntrack_default_on sysctl")
attempted to provide an alternative to the 'default on' behaviour by
adding a sysctl to change it.
However, as Eric points out, the sysctl only becomes available
once the module is loaded, and then its too late.
So we either have to move the sysctl to the core, or, alternatively,
change conntrack to become active only once the rule set requires this.
This does the latter, conntrack is only enabled when a rule needs it.
Reported-by: Eric Dumazet <[email protected]>
Signed-off-by: Florian Westphal <[email protected]>
Signed-off-by: Pablo Neira Ayuso <[email protected]>
Link to the entire commit.
How to revert to the expected behavior?
After disabling ufw or clearing the iptables rules set, create a new SSH session. It will survive a subsequent ufw enable, but might be subject to a random drop out at some point.
This issue will be taken upstream at some point, via the related e-mail list.
EDIT: upstream e-mail thread (contains a work around). Workaround copied here:
echo 1 | sudo tee /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
EDIT 2: upstream proposed patch , which I have tested and reported back.
EDIT 3: 2018.11.06: This has stalled upstream, and I haven't had time to pester them. I'll try to get back to it soon.
EDIT 4: 2019.03.17: I can not reliably reproduce this issue with kernel 5.0.