iptables drops some packets on port 80 and i don't know the cause
We are running a firewall with iptables on our Debian Lenny system. I show you only the relevant entries of our firewall.
Chain INPUT (policy DROP 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 state NEW
Chain OUTPUT (policy DROP 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
LOGDROP all -- * * 0.0.0.0/0 0.0.0.0/0
Some packets get dropped each day with log messages like this:
Feb 5 15:11:02 host1 kernel: [104332.409003] dropped IN= OUT=eth0 SRC=<OWN_IP> DST=<REMOTE_IP> LEN=1420 TOS=0x00 PREC=0x00 TTL=64 ID=18576 DF PROTO=TCP SPT=80 DPT=59327 WINDOW=54 RES=0x00 ACK URGP=0
for privacy reasons I replaced IP Addresses with <OWN_IP> and <REMOTE_IP>
This is no reason for any concern, but I just want to understand what's happening. The web server tries to send a packet to the client, but the firewall somehow came to the conclusion that this packet is "UNRELATED" to any prior traffic.
I have set a kernel parameter ip_conntrack_ma to a high enough value to be sure to get all connections tracked by iptables state module:
sysctl -w net.ipv4.netfilter.ip_conntrack_max=524288
What's funny about that is I get one connection drop every 20 minutes:
06:34:54
06:52:10
07:10:48
07:30:55
07:51:29
08:10:47
08:31:00
08:50:52
09:10:50
09:30:52
09:50:49
10:11:00
10:30:50
10:50:56
11:10:53
11:31:00
11:50:49
12:10:49
12:30:50
12:50:51
13:10:49
13:30:57
13:51:01
14:11:12
14:31:32
14:50:59
15:11:02
That's from today, but on other days it looks like this, too (sometimes the rate varies).
What might be the reason?
Any help is greatly appreciated. kind regards Janning
I have seen a very similar issue on my own systems and I may have an answer as to what is happening.
Apparently, TCP allows the client to send a FIN packet, and you to ACK that FIN packet, but still keep your end of the connection open and push more data through it, sending your own FIN packet at some time in the future. I remember reading that this was implemented at some point in Kernel 2.6 although my memory on this point may be inaccurate/irrelevant.
Many firewalls, including IPTables, don't seem to have implemented this yet or implement it incorrectly and consider the connection closed as soon as they see a FIN followed by an ACK. The result of this is that every so often, my server sends a packet out to a client that the firewall thinks is not part of an established connection and is not new and so drops it.
In practice, I haven't seen any important data in those final few packets that are being dropped, so the effect is a few extra reset packets getting thrown around and some extra connections left in the FIN_WAIT state but no broken or half-loaded pages.
I don't know about the 20 minute timings you are seeing but I would guess that it's a client of some sort that runs regularly every 20 minutes and does a request that causes your server to do the FIN ACK PSH FIN RST sequence.
This post may offer more information about exactly why IPTables drops these packets: http://lists.netfilter.org/pipermail/netfilter/2005-August/062059.html
According to Chris Brenton, it's a mismatch between the timeouts of the client, the server and the firewall for packets in the one-side-closed state.
Does the originating/source IP show up in that log output? If yes does that IP show any valid requests in the http logs? Perhaps a monitoring system of some kind is checking http on your server, since you said it was in consistent intervals. Just throwing stuff out there.