Apache unresponsive (site down) or crashes, MaxRequestWorkers setting reached
This question is about Apache crashing repeatedly, and while it runs, it's unresponsive, i.e. I can't reach the site, so I'm trying to figure out the reason why.
My Owncloud site served by Apache 2.4.7 on Ubuntu 14.04 is not reachable. To check what's wrong, I ran apachectl -S:
~$ sudo apachectl -S
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443 oc.benopp.org (/etc/apache2/sites-enabled/000-default.conf:1)
*:80 127.0.1.1 (/etc/apache2/sites-enabled/000-default.conf:18)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/lock/apache2" mechanism=fcntl
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
This doesn't seem to explain the problem though. The error.log says this:
[Wed Jan 07 19:07:13.384687 2015] [ssl:warn] [pid 1699] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Jan 07 19:07:13.389319 2015] [mpm_prefork:notice] [pid 1699] AH00163: Apache/2.4.7 (Ubuntu) mod_gnutls/0.5.10 PHP/5.5.9-1ubuntu4.5 OpenSSL/1.0.1f configured -- resuming normal operations
[Wed Jan 07 19:07:13.389359 2015] [core:notice] [pid 1699] AH00094: Command line: '/usr/sbin/apache2'
[Wed Jan 07 19:08:01.788631 2015] [mpm_prefork:error] [pid 1699] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
I don't understand how the Max requests settings can be reached with only me using the site. Am I right in assuming that while I'm the only one trying to access Owncloud to download some files or change settings, that should never cause this amount of requests?
To take a closer look, I activated mod_status. Unfortunately I don't know what to make of the readout, because I only taught myself how to setup Apache to run my Owncloud, and I'm new to this. It seems a bit long to post the server-status page readout here inline, so I made a paste: http://pastie.org/private/aj7zduzhrt8sayjb4hncg
So, all the processes are "sending reply", except I don't know to what, because I'm not sending any requests, and the server is actually disconnected from the internet right now. In the documentation it is explained how to look for problems that are causing excessive workloads, but I'm having trouble getting Top to show me something I can use. It does seem though that CPU-load really isn't it, there are no high loads to be seen. Can you tell what the problem is from this readout?
EDIT2: another crash happened a few hours ago, here's a part that seems relevant from the error.log:
beginning of file:
ProblemType: Crash
Architecture: amd64
CrashCounter: 1
Date: Tue Jan 6 13:50:05 2015
DistroRelease: Ubuntu 14.04
ExecutablePath: /usr/sbin/apache2
ExecutableTimestamp: 1406039884
ProcCmdline: /usr/sbin/apache2 -k start
ProcCwd: /etc/apache2
ProcEnviron:
PATH=(custom, no user)
LANG=C
ProcMaps:
7fd2a4000000-7fd2a8000000 rw-s 00000000 00:04 12964 /dev/zero (deleted)
7fd2a8000000-7fd2a8021000 rw-p 00000000 00:00 0
7fd2a8021000-7fd2ac000000 ---p 00000000 00:00 0
7fd2aced8000-7fd2acedd000 r-xp 00000000 08:01 266060 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7fd2acedd000-7fd2ad0dc000 ---p 00005000 08:01 266060 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7fd2ad0dc000-7fd2ad0dd000 r--p 00004000 08:01 266060 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
and so on and so on, until end of file:
ProcStatus:
Name: apache2
State: R (running)
Tgid: 2807
Ngid: 0
Pid: 2807
PPid: 1838
TracerPid: 0
Uid: 33 33 33 33
Gid: 33 33 33 33
FDSize: 64
Groups: 33
VmPeak: 525052 kB
VmSize: 524944 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 23680 kB
VmRSS: 23680 kB
VmData: 85112 kB
VmStk: 136 kB
VmExe: 592 kB
VmLib: 74868 kB
VmPTE: 712 kB
VmSwap: 0 kB
Threads: 1
SigQ: 1/29106
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000004000
SigIgn: 0000000001001000
SigCgt: 000000018c0042eb
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 21
nonvoluntary_ctxt_switches: 34
Signal: 11
Uname: Linux 3.13.0-43-generic x86_64
UserGroups:
CoreDump: base64
then the CoreDump I deleted.
Solution 1:
This sounds like the following bug posted on launchpad: https://bugs.launchpad.net/ubuntu/+source/php-apcu/+bug/1422484
The short version is that it seems that there is a problem with the APC User Cache for PHP 5, at least the version packaged with Ubuntu 14.04.
The fix seems to be to purge the php5-apcu package and then restart apache:
sudo apt-get purge php5-apcu
sudo service apache2 restart
I have just done this so I am not sure if it will actually fix this long term. I was also seeing APC errors in the error log.
If you found a different solution, I would love to hear it.