iptables: match only the first packet of established TCP-connection

Solution 1:

Forget IPTables. You can simply use mod_security with nolog action. Something like this (untested):

SecRule REQUEST_URI "^/w00tw00t\.at\.ISC\.SANS\.DFind" phase:1,nolog,deny,id:1000

Or you can create a dummy virtualhost with separate logs, which just denies all requests and configure it as first. Clients which would not provide a hostname or provide unknown hostname would always end there.