Trying to make iptables stateless is causing unforeseen filtering

You have a rule which blocks all incoming traffic:

-A INPUT -j REJECT

And you stop connection tracking, so the rule to accept established connections' packets doesnt work anymore:

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

So your DNS packet goes out, is not tracked, and is then rejected by the first rule.

You need to enable tracking for the second rule to work, or add rules to allow incoming traffic from "good" sources.

Linux connection tracking doesn't just track TCP connections, it also tracks UDP psuedo-connections and various other things.

Without the notrack rules what happens is.

• Your system sends a DNS request, this creates a connection tracking entry.
• The DNS server generates a reply
• The reply matches your "accept established and related" rule.

With the notrack rule what happens is

• Your system sends a DNS request, but does not create a connection tracking entry.
• The DNS server generates a reply
• The reply does not match your "accept established and related" rule.
• The reply falls through and hits your catch-all reject rule.

So how to fix this? Either you have to explicitly allow response traffic without relying on your state tracking or you have to get more selective about your notrack rules.

For DNS explicitly allowing response traffic is probablly reasonable. You know what your DNS server is and you probablly trust it.

If your server needs to access resources on the internet in general then going compeletely without connection tracking means going with a far less effective firewall.

I expect you could get much of the performance benefit with far less pain by only applying notrack to traffic to and from your webserver. e.g.

iptables -t raw -I OUTPUT -p tcp --sport http -j NOTRACK
iptables -t raw -I PREROUTING -p tcp --dport http -j NOTRACK