Invalid argument: AH03139: ap_queue_pop failed
Short version, I upgraded from Fedora Server 23 to Fedora Server 24, and now my apache (httpd-2.4.23-4.fc24.x86_64) error_log is filling up with these messages:
[Mon Nov 28 20:30:39.486187 2016] [mpm_worker:crit] [pid 9973:tid 140499117635328] (22)Invalid argument: AH03139: ap_queue_pop failed
[Mon Nov 28 20:30:39.486197 2016] [mpm_worker:crit] [pid 9973:tid 140499117635328] (22)Invalid argument: AH03139: ap_queue_pop failed
(It set off disk space alerts and I pruned error_log when it hit 1.8Gb!)
I am using the MPM worker, with these settings:
ServerLimit 60
MaxRequestWorkers 1500
MinSpareThreads 10
MaxSpareThreads 25
MaxRequestsPerChild 10000
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 29966
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 29966
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Any advice would be appreciated. :)
Apache Server Status for example.com (via x.x.x.x)
Server Version: Apache/2.4.23 (Fedora) OpenSSL/1.0.2j-fips PHP/5.6.28
Server MPM: worker
Server Built: Jul 18 2016 15:38:14
Current Time: Wednesday, 30-Nov-2016 12:48:45 AST
Restart Time: Tuesday, 29-Nov-2016 18:21:57 AST
Parent Server Config. Generation: 3
Parent Server MPM Generation: 2
Server uptime: 18 hours 26 minutes 48 seconds
Server load: 6.60 6.48 6.33
Total accesses: 2278650 - Total Traffic: 20.6 GB
CPU Usage: u488.03 s82.94 cu4.36 cs14.77 - .889% CPU load
34.3 requests/sec - 325.3 kB/second - 9.5 kB/request
146 requests currently being processed, 79 idle workers
_W_K_KKKCKKKKK_K__KKK___CKK_KKK__K_K___K___CKK_W_K_K__C_KK___K_K
KKCK_CK__CKKWCK__WKKKKKKW_WWKKKKC_____K__K_KKKKKKKWK_K_KK__KK...
......................KKKKKKKK_K_KKC____KKKCKKKKK__KCKKKKC_KC__K
_KKKK_CKK_KK__K_KKKK___RKCCK_K_KK_KC_K_KKK_K__W_K_KK__KKKK......
................................................................
................................................................
................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
EDIT: In the end, I believed the server was overall unstable and scrapped it, building a new one from scratch. I was never able to solve the issue.
It seems like your system limits are not high enough, causing Apache processes to have issues. You might want to increase the values of max user processes, pending signals and open files. Here some preferable values:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1546671
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 102400
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1546671
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited