HAProxy stick table rate expiration
I am using HAProxy on Ubuntu to perform some rate limiting. Here are the most important fragments from my config:
global
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
frontend http-listener
bind *:80 accept-proxy
http-request track-sc1 src table st_ip_req_rate
http-request deny deny_status 429 if { sc_http_req_rate(1) gt 100 }
http-request deny deny_status 429 if { sc_http_err_rate(1) gt 200 }
default_backend no-match-backend
backend st_ip_req_rate
stick-table type ip size 100k expire 24h store http_req_rate(10s),http_err_rate(24h)
backend no-match-backend
http-request deny deny_status 503
I did not include my actual backends here but within this example my expectations would be:
- requests are usually answered with a status 503
- the 101st request within a sliding window of 10 seconds is answered with a status 429
- the first request after 200 requests within a sliding window of 24 hours that have been answered with a 4xx status code is answered with a status 429
I expect the sliding window to work for the sc_http_req_rate
because the example this blog post also uses a rate over 10s with an expire 30s
. However the requests are denied with a 429 after 100 requests. When I look into the stick table like such:
$ echo "show table st_ip_req_rate" | sudo socat unix:/run/haproxy/admin.sock -
# table: st_ip_req_rate, type: ip, size:102400, used:10
0x7f133c01b8d8: key=192.1.2.3 use=0 exp=86389546 http_req_rate(10000)=639 http_err_rate(86400000)=0
...
I can see that the http_req_rate does not decline.
Am I wrong to expect different behavior or is this a bug? I checked the link below and no known bug seemed to match at the time (25.03.2021).
$ haproxy -v
HA-Proxy version 2.2.11-1ppa1~focal 2021/03/18 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.11.html
Running on: Linux 5.4.0-67-generic #75-Ubuntu SMP Fri Feb 19 18:03:38 UTC 2021 x86_64
The answer from this question suggests to use table_http_req_rate
for a similar problem. This would not apply here, as my request rates also don't decline if there are no requests for > 10s.
Solution 1:
I've hit this as well. It seems to be regression. Switching to 2.3.X (specifically 2.3.7) resolved the issue for me. If you're running it from docker using lts-apline
as I was, you'll have to switch to a different image for now.