Haproxy Falling over under high volume
I am having some issues with HAProxy configuration. I have been playing around with it to try and make it more resilient to high server loads and Denial of Service. However, I felt it was working fine until suddenly I was victim of a (D)DoS attack - Haproxy was reporting the backend as down even though I could still access it fine via the direct port.
Could somebody please check my HAProxy config and see if there is somewhere I am messing up or why I would be experiencing this.. I just can't seem to understand why this is happening.
Thanks in advance (and after of course) .
global
# Global Max Connections
maxconn 20000
# Various Other Settings
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.stat mode 600 level admin
stats timeout 5m
chroot /usr/share/haproxy
daemon
# User Settings
user haproxy
group haproxy
defaults
# Default configuration settings for Haproxy
retries 2
maxconn 19500
timeout server 10s
timeout client 10s
timeout queue 10s
timeout connect 10s
timeout http-request 10s
# Error files
errorfile 503 /etc/phpconf/haErrors/503.http
frontend Connection_Handler
default_backend Primary
bind :80
mode http
option forwardfor
option http-server-close
maxconn 20000
# Check if cookie exists
#acl cookie_set hdr_sub(cookie) authorized=1
# If cookie doesn't exist try and set it
#redirect prefix * set-cookie authorized=1 if !cookie_set
# If the cookie is still not set, send it to blocked backend
#use_backend Cookie_Block if !cookie_set
## (D)DoS Mitigation ##
# Setup stick table
stick-table type ip size 1m expire 10m store gpc0
# Configure the DoS src
acl src_DoS src_get_gpc0(Connection_Handler) gt 0
# Use DoS tarpit if src_DoS
use_backend DoS_Tarpit if src_DoS
# If not blocked, track the connection
tcp-request connection track-sc1 src if ! src_DoS
listen Statistics_Engine
mode http
bind XX.XXX.XX.XX:9012
stats enable
stats uri /admin?stats=true
stats auth admin:Password
stats hide-version
stats refresh 2s
#stats scope # Add this option to provide stats for a singular backend
backend Primary
# Option Configs
option httpclose
option redispatch
option abortonclose
## (D)DoS Mitigation ##
# The following table is recording the IP, connection rate and bytes out rate
stick-table type ip size 200k expire 10s store conn_rate(5s)
# Track request and enforce rules
tcp-request content track-sc2 src
# Mark as abuse if exceeding connection rate
acl conn_rate_abuse sc2_conn_rate gt 80
# Mark as abuse if over X bytes
acl data_rate_abuse sc2_bytes_out_rate gt 200000
# Set ACL rule to enforce on frontend
acl mark_as_DoS sc1_inc_gpc0 gt 0
# Block connections marked as DoS
tcp-request content reject if conn_rate_abuse mark_as_DoS
#tcp-request content reject if data_rate_abuse mark_as_DoS
# Configure Server
mode http
option forwardfor
server Primary_HTTP 0.0.0.0:1080 check addr 127.0.0.1 port 80 inter 3000 rise 2 fall 3 maxconn 20000
#fullconn 1024
backend Conn_Tarpit
# Tarpit for connections
mode http
timeout tarpit 20s
reqitarpit .
errorfile 503 /etc/phpconf/haErrors/tarpit_503.txt
backend Cookie_Block
# Block connections that will not take on a cookie
mode http
reqdeny .
errorfile 503 /etc/phpconf/haErrors/503_cookie.txt
backend DoS_Tarpit
# Tarpit for suspected attacks
log 127.0.0.1 local1 info
timeout tarpit 10s # Tarpit for 10 seconds
errorfile 500 /etc/phpconf/haErrors/500_DoS.txt
mode http
reqitarpit .
I see nothing obviously wrong in your config, you seem to have already taken care of correctly tuning your settings (especially maxconn). Is conntrack loaded on this machine ? The connection table might be full, preventing checks and connections from establishing to the server.
Also, have you checked how many concurrent connections were sent to the server ? It's possible that the server is alternatively going up and down due to the load.
Check the kernel log messages for any unexpected error.