iptables: building a rule set against abuse for DNS amplification attacks

Question 1:

The string does not match because the the "." is not included in the packet. A DNS packet does not contain a "hostname" as such but "labels". In the packet, every part of the domain name is a label, prefixed by the number of bytes for the label.

So "isc.org" translates to:

isc: 03 69 73 63
org: 03 6f 72 67

Or in the packet:

03697363036f7267

Every label is limited to 63 bytes, the whole name is limited to 255 bytes.

It's explained in the DNS RFC:

https://www.rfc-editor.org/rfc/rfc1035#section-2.3.4

https://www.rfc-editor.org/rfc/rfc1035#section-4.1.2

Question 2:

You need to enable the net.netfilter.nf_conntrack_acct flag to use the conntrack option (see iptables manpage). But I don't think it's wise to use it like that. There will always be legitimate answers that are large packets.

Perhaps you're better off using the hashlimit extension. It was already mentioned:

https://lists.dns-oarc.net/pipermail/dns-operations/2012-October/009321.html