netstat and ip_conntrack connection count differ by order of magnitude. Why?

Solution 1:

Conntrack module remembers recent connections for X seconds before they finally expire. This, in my understanding, is because iptables has several other modules that can utilize this information: for example, if you want to ban some IP address if it makes X new connections during some time frame.

netstat, on the other hand, shows real-time information and is not interested about ancient history.

Have you increased maximum amount of entries in conntrack table? With a recent-ish kernel, what does

sysctl net.ipv4.netfilter.ip_conntrack_max

... or with some older kernel,

sysctl net.ipv4.ip_conntrack_max

return to you? You may raise that value permanently via /etc/sysctl.conf or temporarily (until next reboot) via sysctl -w net.ipv4.ip_conntrack_max

Solution 2:

We stumbled across this case when containers (docker) were in use.

Not sure if it helps in your case or not, but it looks like netstat -nat on the host OS will only show connections intended for the host's networking stack whereas conntrack -L will show information for both the host and all its containers.

If you run netstat -nat from inside the container involved in the connection reported by conntrack -L, you should see the connection information listed there.