Percona-server time out on /etc/init.d/mysql start

The display of the error log looks quite normal

Socket file was created MySQL Replication started just fine /usr/sbin/mysqld: ready for connections. appears

Especailly since you see /usr/sbin/mysqld: ready for connections., you should be able to connect to mysql which you just stated that you can.

Your mysqld process is fine.

The error may be coming from /etc/init.d/mysql but not before mysqld has done everything else it needed to do.

If you look inside /etc/init.d/mysql there are two lines

[root@***** init.d]$ cat mysql | grep -n "&" | grep "pid-file"
313:      $manager --user=$user --pid-file=$pid_file >/dev/null 2>&1 &
327:      $bindir/mysqld_safe --datadir=$datadir --pid-file=$server_pid_file $other_args >/dev/null 2>&1 &

If /etc/init.d/mysql timed out, it definitely occurred after these two lines.

Here is the code that checks for the pid file

wait_for_pid () {
  verb="$1"
  manager_pid="$2"  # process ID of the program operating on the pid-file
  i=0
  avoid_race_condition="by checking again"
  while test $i -ne $service_startup_timeout ; do

    case "$verb" in
      'created')
        # wait for a PID-file to pop into existence.
        test -s $pid_file && i='' && break
        ;;
      'removed')
        # wait for this PID-file to disappear
        test ! -s $pid_file && i='' && break
        ;;
      *)
        echo "wait_for_pid () usage: wait_for_pid created|removed manager_pid"
        exit 1
        ;;
    esac

    # if manager isn't running, then pid-file will never be updated
    if test -n "$manager_pid"; then
      if kill -0 "$manager_pid" 2>/dev/null; then
        :  # the manager still runs
      else
        # The manager may have exited between the last pid-file check and now.
        if test -n "$avoid_race_condition"; then
          avoid_race_condition=""
          continue  # Check again.
        fi

        # there's nothing that will affect the file.
        log_failure_msg "Manager of pid-file quit without updating file."
        return 1  # not waiting any more.
      fi
    fi

    echo $echo_n ".$echo_c"
    i=`expr $i + 1`
    sleep 1
  done

  if test -z "$i" ; then
    log_success_msg
    return 0
  else
    log_failure_msg
    return 1
  fi
}

wait_for_pid() is called after the launch of mysqld_safe

  # Give extra arguments to mysqld with the my.cnf file. This script
  # may be overwritten at next upgrade.
  pid_file=$server_pid_file
  $bindir/mysqld_safe --datadir=$datadir --pid-file=$server_pid_file $other_args >/dev/null 2>&1 &
  wait_for_pid created $!; return_value=$?

In the worst case, mysqld is running without a pid file. In light of this, service mysql stop and /etc/init.d/mysql stop may not work correctly as it checks for the pid file to know the process id of mysqld.

Without a pid file, the proper way to shutdown mysqld is

# mysqladmin -uroot -h127.0.0.1 --protocol=tcp -p shutdown

CAVEAT

This is not that local of a phenomenon. I have seen this occur with standard MySQL binaries as well.


It appears that the init script checks for a running MySQL/Percona server on Ubuntu by using mysqladmin --defaults-file=/etc/mysql/debian.cnf ping (see line 27 and 77 of /etc/init.d/mysql of the in this case percona-server-server-5.5 package version 5.5.28-rel29.2-360.precise). This works in a default new installation, but when copying over data files in setting up replication from another machine, the user configured in debian.cnf does not match anymore.

As a result, the mysqladmin command will fail and the init script will report the service failed, yet it just runs fine as you can see in the logs.

One solution is to re-create the MySQL user the init scripts expect. On the master, just add the user like this (seems to have all privileges by default?!):

GRANT ALTER, ALTER ROUTINE, CREATE, CREATE ROUTINE, CREATE TEMPORARY TABLES,
CREATE USER, CREATE VIEW, DELETE, DROP, EVENT, EXECUTE, FILE, INDEX, INSERT,
LOCK TABLES, PROCESS, REFERENCES, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE,
SELECT, SHOW DATABASES, SHOW VIEW, SHUTDOWN, SUPER, TRIGGER, UPDATE
ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY PASSWORD 
*replacethiswithaknownhash' WITH GRANT OPTION;

and create/modify the /etc/mysql/debian.cnf file:

[client]
host     = localhost
user     = debian-sys-maint
password = yourpass
socket   = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host     = localhost
user     = debian-sys-maint
password = yourpass
socket   = /var/run/mysqld/mysqld.sock
basedir  = /usr

You may need to wait to have the replication process catched up and then try to restart.