firewalld rich rules don't drop incoming traffic (CentOS 8 behind a NAT)
Post-Solving Edit
The reason this was so hard to solve at the firewall level was that it wasn't a firewall problem. Something @tom-yan said in chat made me revisit the script which pulls out the IP Addresses. It seems that sed was pumping out the entire access_log file (rather than what I was telling it - as it did in testing). I replaced it with grep and the script is now exporting what I would expect.
Which makes this yet another example of check your assumptions.
Thank you to Tom Yan & @pillsbury-it-doughboy for your efforts. I appreciate it.
(Since Tom's is the only surviving answer, I'm going to mark his as the correct one.)
G'day,
I've got a small development server at my office with port 80 and 443 port forwarded from the modem.
To restate the title in the form of a question:
Why doesn't the firewall drop the incoming traffic from the IPv4 addresses that are listed in the rich rules.
Background: As you would expect, Bots & Baddies™ are looking for various things that a) don't exist, and b) would be bad if they they got into if they did exist. So I have a script that pulls IP addresses from Apache logs, which then end up in the firewall after they've been curated by me.
However, the firewall isn't dropping the connections. Addresses that are added on previous days are present in the new rules to be added and the logs again on subsequent days.
Details: The command that puts in the rich rules is this:
firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='165.227.87.0/24' reject"
However some rules have extra info:
rule family="ipv4" source address="162.159.244.38" log prefix="Shodan.io" reject
While some are simply:
rule family="ipv4" source address="42.224.165.0/24" reject
(those last two are rules copied from the output of firewall-cmd --list-all)
As shown below, the active zone is public, which isn't explicitly noted in the rule as I read that without it specified, it applies to the active zone.
While researching this, I realised that I might be in an unusual situation with the Modem port forwarding to the machine, rather than having it in a DMZ or hosted externally. The Apache logs show the internet facing IP addresses for the http/s client machines, and I've been assuming that these IPs are the address that presented to the firewall.
(netstat -tn does show a current connection to an external IPv4 address, but I can't establish if that's inbound or outbound.)
Warning: ALREADY_ENABLED: rule family='ipv4' source address='201.159.155.0/24' reject was a frequent response until I added logging notes into the rule, now it just happily accepts multiple reject rules for the same subnet.
Since the first time I wrote this question on serverfault.com (and was then told it wasn't ok to ask this there), I've had time to add the aforementioned logging, and have now been been to establish this:
[user@server ~]# firewall-cmd --list-rich-rules | grep 194.195.251.
rule family="ipv4" source address="194.195.251.0/24" log prefix="Sun May 23 21:23:50 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Mon May 24 23:58:51 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Tue May 25 21:25:27 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Wed May 26 21:25:57 UTC 2021" reject
And this:
[user@server ~]# firewall-cmd --list-rich-rules | grep 192.241.196.
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Mon May 24 23:59:25 UTC 2021" reject
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Tue May 25 21:26:02 UTC 2021" reject
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Wed May 26 21:26:31 UTC 2021" reject
So clearly, this isn't dropping traffic. (The date/times are the dates added to the firewall.)
In case it's helpful, here's the top part of firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eno1
sources:
services: cockpit dhcpv6-client http https ssh
ports: [redacted integer]/tcp [redacted integer]/tcp [redacted integer]-[redacted integer]/udp [redacted integer]-[redacted integer]/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
The rich rules then follow. And because why not, here's the active zone query:
[user@server ~]# firewall-cmd --get-active-zone
libvirt
interfaces: virbr0
public
interfaces: eno1
Other things to note:
- There is only one Ethernet port on the box.
- The firewall is reloaded after adding new rules (
firewall-cmd --reload) - The server reboots every night
- This server was CentOS Stream, but I converted it to CentOS 8
- The firewall is active, enabled and functional (
firewall-cmd --statereturns 'running') - I'm aware that
firewall-cmd --complete-reloadexists, but it terminates all connections and that's not what I need. - Some things from the conf file:
FirewallBackend=nftablesIPv6_rpfilter=yesRFC3964_IPv4=yes
Any ideas?
( List of Rich Rules: https://pastebin.com/qXFMBvqh )
-- EDIT --
Under the banner of "This is ridiculous", I put another local IP in the firewall (without --permanent) to test, then ran curl from the CLI. The pre-adding test worked fine, the post-adding test gave me curl: (7) Failed connect to **[redacted]**:80; Connection refused. With that behaving as expected, I cannot work out WHY these other things are not.
Have now turned on official logging (firewall-cmd --set-log-denied=all)
-- EDIT 2 -- More info on the scripts in question:
- The first script parses the Apache logs and establishes a list of IP addresses. Upon that, it puts the
firewall-cmdadding commands into a.txtfile. (It also looks up Geo-location information, but that's out of scope here.) - The second script reads the
.txtfile, and executes thefirewall-cmdadd commands, then runs the firewall reload (--reload)
This allows the .txt file to build up over the weekend if I'm AFK. I inspect the GEO location information prior to running the second script to ensure that any IPs in the same country are more closely inspected to see whether they should be removed from the list.
-- EDIT 3 -- Logs shows this:
Jun 21 09:47:42 <SERVER_NAME> kernel: STATE_INVALID_DROP: IN=eno1 OUT= MAC=a8:a1:59:3a:03:14:20:b0:01:c0:a5:26:08:00 SRC=183.158.154.128 DST=<SERVER_INTERNAL_IP> LEN=40 TOS=0x00 PREC=0x00 TTL=44 ID=38952 DF PROTO=TCP SPT=54750 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
There are four of these entries, all at the same millisecond. Interestingly, that IP address is not in the firewall rules at all.
Solution 1:
See if the following workaround makes any difference. Run the following commands as root / with sudo (every time after reloading rules; shouldn't be necessary after adding new rules without --permanent afterwards though):
(Btw, make sure you bare in mind that rules added without --permanent will be gone after reloading, just like the following.)
nft add chain inet firewalld filter_IN_public_black
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_log
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_deny
nft insert rule inet firewalld filter_INPUT jump filter_IN_public_black
(Since typical table flush does NOT remove chains, but only rules, added to a table, you probably can't use create instead of add for the filter_IN_public_black chain to determine whether this needs to be applied again. So just check with e.g. nft list ruleset when in doubt. But of course, firewalld might enumerate all chains in its tables and delete them one by one upon reload. Do some tests if you want to know.)
P.S. This override will be applied to ALL interfaces / "zones" (with log and deny rules added, before or after the override is applied if without --permenant, for the public zone). If desired, further customize it to make it less "aggressive" (e.g. add an iifname match for the jump to filter_IN_public_black that is inserted to filter_INPUT).