Mysterious error with php5-fpm

I'm having a very difficult case with latest php5-fpm (from debphp) and nginx 1.4.3 on Ubtuntu 12.04LTS.

In short, I cannot start php5-fp at all, I keep getting this error in syslog:

kernel: [1213013.564441] init: php5-fpm main process (7357) terminated with status 78
kernel: [1213013.564496] init: php5-fpm main process ended, respawning
..... (repeated several times)
kernel: [1213014.143911] init: php5-fpm respawning too fast, stopped

My /var/log/php5-fpm.log is not populated at all. There is no php5-fpm.sock and php5-fpm.pid in /run (or /var/run).

My /etc/php5/fpm/php-fpm.conf is below: (this file doesn't seem to be loaded now as I tried to put some bogus info into it but didn't cause any change)

;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;

include=/etc/php5/fpm/*.conf

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
; Pid file
; Note: the default prefix is /var
; Default Value: none
pid = /run/php5-fpm.pid
catch_workers_output = yes


; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = debug

; To configure the pools it is recommended to have one .conf file per
; pool in the following directory:
include=/etc/php5/fpm/pool.d/*.conf
include=/var/www/vhosts/mysite.com/w/w/w/www/config/fpm-pool.conf

Permission checking:

ls -l /etc/php5/fpm/php-fpm.conf

-rwxrwxrwx 1 root root 4641 Nov  6 01:37 /etc/php5/fpm/php-fpm.conf

My include=/var/www/vhosts/mysite.com/w/w/w/www/config/fpm-pool.conf (this file does not seem to be loaded anymore)

[www.mysite.com]
    listen                 = /run/php5-fpm.sock
#    listen                = 127.0.0.1:9000
    listen.backlog         = -1
    listen.allowed_clients = 127.0.0.1
    listen.owner           = www.mysite.com
    listen.group           = mysite.com
    listen.mode            = 0666

    user  = www.mysite.com
    group = mysite.com

    pm                   = dynamic
    pm.max_requests      = 0
    pm.max_children      = 15
    pm.start_servers     = 2
    pm.min_spare_servers = 1
    pm.max_spare_servers = 3

    pm.status_path       = /php_pool_wwww.mysite.com_status
    ping.path            = /www.mysite.com_ping
    ping.response        = www.mysite.com_pong

    request_terminate_timeout = 30
    request_slowlog_timeout   = 20
    slowlog                   = /var/www/vhosts/mysite.com/w/w/w/www/logs/php-slow.log

    rlimit_files = 131072
    rlimit_core = unlimited

     chroot = /var/www/vhosts/mysite.com/w/w/w/www/
    ; Chdir to this directory at the start. This value must be an absolute path.
    ; Default Value: current directory or / when chroot
    ; chdir = /htdocs

    catch_workers_output = yes

    env[HOSTNAME] = $HOSTNAME
    env[TMP]      = /tmp
    env[TMPDIR]   = /tmp
    env[TEMP]     = /tmp

    security.limit_extension = .php
    ;   php_value/php_flag             - you can set classic ini defines which can
    ;                                    be overwritten from PHP call 'ini_set'.
    ;   php_admin_value/php_admin_flag - these directives won't be overwritten by
    ;                                     PHP call 'ini_set'
    php_flag[display_errors]            = on
    php_admin_value[error_log]          = /logs/php_err.log
    php_admin_flag[log_errors]          = on
    php_admin_value[memory_limit]       = 100M
    php_value[max_execution_time]       = 300

Permission checking:

ls -l /var/www/vhosts/mysite.com/w/w/w/www/config/fpm-pool.conf

-rwxr-xr-x 1 www.mysite.com mysite.com 1830 Nov  6 01:05 var/www/vhosts/mysite.com/w/w/w/www/config/fpm-pool.conf

User/Group Permission checking:

groups www.mysite.com

www.mysite.com : www.mysite.com www-data mysite.com

I don't think this has anything to do with nginx because php5-fpm itself cannot start at all right now anyhow, but for the sake of having all info: nginx is also set to use same user/group

Solution 1:

The process that upstart was watching died, but the pool workers are still running. To fix it, run:

sudo killall php-fpm
sudo start php5-fpm

The php5-fpm job will repeatedly exit with status 78 until all of the old worker threads are dead. Note that the server is typically functional in this state, even though upstart isn't managing the service. When it eventually dies, though, it won't automatically respawn, so it should be fixed ASAP. A simple reboot will also fix the problem, but will result in downtime.

Solution 2:

I also got these errors in /var/log/syslog:

init: php5-fpm main process (7357) terminated with status 78
init: php5-fpm main process ended, respawning
init: php5-fpm respawning too fast, stopped

Notes

• I tried using these tips but found no difference.

• Adding php5-fpm as a service didn't work using:

    update-rc.d php-fpm defaults
  

  even if I only triggered it only at specific run levels. This can be removed again using:

    update-rc.d -f php-fpm remove
  

• Without installing as a service, php5-fpm would still attempt to start (and fail)

• Running service --status-all | grep php always showed php5-fpm as stopped, even after my fix below.

Fix

Using the kill tip from @Zenexer though I got the following script to work:

#!/bin/bash
# start-php5-fpm-reboot.sh
# Add a delay, otherwise the script fires too soon
sleep 30

# Kill all php5 children
killall php5-fpm

# Start the php5-fpm service
/usr/bin/service php5-fpm start

Place it in e.g. /etc/php5/fpm, ensure root has exectuable permissions, then add it as a cron job by using sudo crontab -e and adding the following line:

@reboot /etc/php5/fpm/start-php5-fpm-reboot.sh