HAProxy reload - old processes never terminated
I have HAProxy setup in TCP mode, with client/server/connect timeout of 120s.
When I reload the configuration too fast, I sometime end up with multiple processes. By design, this is expected, so all established connections are drained.
My issue is that they not never terminated, even though all connections are closed.
ps aux | haproxy
haproxy 12483 0.0 0.1 103748 1084 ? Ss 20:45 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 12405
haproxy 12485 0.0 0.1 103748 1088 ? Ss 20:45 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 12405
haproxy 12487 0.0 0.1 103748 1084 ? Ss 20:45 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 12405
haproxy 25115 0.0 0.1 103748 1084 ? Ss 21:26 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 12488
netstat -pant | grep haproxy
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 25115/haproxy
tcp 0 0 0.0.0.0:1936 0.0.0.0:* LISTEN 25115/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 25115/haproxy
I waited longer than the timeout of 120s. I don't understand what is holding them.
The following lsof for one of those old process show that there are still some FD for TCP LISTEN
# lsof -p 12483
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 12483 haproxy cwd DIR 202,1 4096 2 /
haproxy 12483 haproxy rtd DIR 202,1 4096 2 /
haproxy 12483 haproxy txt REG 202,1 4381869 412355 /usr/local/sbin/haproxy
haproxy 12483 haproxy mem REG 202,1 62864 396140 /lib64/libnss_files-2.17.so
haproxy 12483 haproxy mem REG 202,1 126288 396526 /usr/lib64/libselinux.so.1
haproxy 12483 haproxy mem REG 202,1 141760 396148 /lib64/libpthread-2.17.so
haproxy 12483 haproxy mem REG 202,1 89312 396076 /lib64/libgcc_s-4.8.2-20140120.so.1
haproxy 12483 haproxy mem REG 202,1 98720 396150 /lib64/libresolv-2.17.so
haproxy 12483 haproxy mem REG 202,1 13224 396957 /lib64/libkeyutils.so.1.5
haproxy 12483 haproxy mem REG 202,1 43768 396966 /lib64/libkrb5support.so.0.1
haproxy 12483 haproxy mem REG 202,1 19512 396128 /lib64/libdl-2.17.so
haproxy 12483 haproxy mem REG 202,1 170784 396962 /lib64/libk5crypto.so.3.1
haproxy 12483 haproxy mem REG 202,1 12744 396594 /usr/lib64/libcom_err.so.2.1
haproxy 12483 haproxy mem REG 202,1 937952 396964 /lib64/libkrb5.so.3.3
haproxy 12483 haproxy mem REG 202,1 273672 396958 /lib64/libgssapi_krb5.so.2.2
haproxy 12483 haproxy mem REG 202,1 486512 396073 /lib64/libfreebl3.so
haproxy 12483 haproxy mem REG 202,1 2000552 396122 /lib64/libc-2.17.so
haproxy 12483 haproxy mem REG 202,1 1967496 400756 /lib64/libcrypto.so.1.0.1j
haproxy 12483 haproxy mem REG 202,1 445424 400761 /usr/lib64/libssl.so.1.0.1j
haproxy 12483 haproxy mem REG 202,1 88568 396529 /lib64/libz.so.1.2.7
haproxy 12483 haproxy mem REG 202,1 36856 396126 /lib64/libcrypt-2.17.so
haproxy 12483 haproxy mem REG 202,1 152376 396115 /lib64/ld-2.17.so
haproxy 12483 haproxy 0u 0000 0,9 0 5420 anon_inode
haproxy 12483 haproxy 4u IPv4 1435667 0t0 TCP *:http (LISTEN)
haproxy 12483 haproxy 5u IPv4 1435668 0t0 TCP *:https (LISTEN)
haproxy 12483 haproxy 6u IPv4 1435673 0t0 TCP *:jetcmeserver (LISTEN)
Just found solution here: https://discourse.haproxy.org/t/tcp-connection-keep-old-process-up-on-soft-reload/2905
Configure hard-stop-after
in haproxy.cfg
global
hard-stop-after 30s
This happened to me too just a few days ago... No reasonable answer, probably, the process never finished because of connections still using it all the time. I have 2 HaProxy's and, this situation never happened in the secondary one as it does not have connections during normal operation.
I issued a SIGTERM command, or you can just KILL the old PID and you are fine.
You can just get the old PID from the HaProxy status page. Refreshing several times I was seeing the old and new process randomly.
After killing the old one, the new process was the only one responding the requests.
:)