Mysterious HAProxy request errors

Solution 1:

Have you looked at dmesg? A common problem with Proxy servers is to hit the max Linux connection tracking since each request is using to connections. If this is the case you will see ip_conntrack: table full, dropping packet. in dmesg. You can see the current count and raise it via sysctl or proc:

[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_max
131072
[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_count
185

You can also bypass connection tracking with the NOTRACK target, i.e.:

sudo /sbin/iptables -t raw -A PREROUTING -p tcp --sport 80 -j NOTRACK

Keep in mind that it is a security risk to disable tracking though, you don't want to do it unless you are already behind a stateful firewall.

Can you post the errors you are seeing?

Solution 2:

I suspect Michael is only seeing them on the stats web page. Most likely those errors are simply due to port scans. If your haproxy configuration has "option dontlognull", you won't get any log for them, but still they are correctly accounted for. You'd see them in the "show errors" if they were invalid HTTP request and this does not seem to be the case. Don't worry for a few per second, it's almost nothing once you've figured that they are just port scans. Some heavy sites are getting between hundreds and thousands a second !

@Tim: recent versions have fixed an issue that older ones had which caused some of these errors not to be reported (they were incidently accounted in the backend where they're not retrieved). I'm sure it's just that.