Percona XtraBackup: Getting an error regarding "Too many open files", how can I solve this issue?
I'm trying to backup using Percona XtraBackup and getting the following error:
[root@ads-dev1 ~]# innobackupex /root/db
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
140722 11:52:16 innobackupex: Starting mysql with options: --unbuffered --
140722 11:52:16 innobackupex: Connected to database with mysql child process (pid=15930)
140722 11:52:22 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".
innobackupex: Using mysql Ver 14.12 Distrib 5.0.95, for redhat-linux-gnu (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
innobackupex: Created backup directory /root/db/2014-07-22_11-52-22
140722 11:52:22 innobackupex: Starting mysql with options: --unbuffered --
140722 11:52:22 innobackupex: Connected to database with mysql child process (pid=15953)
140722 11:52:24 innobackupex: Connection to database server closed
140722 11:52:24 innobackupex: Starting ibbackup with command: xtrabackup_51 --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/root/db/2014-07-22_11-52-22 --tmpdir=/tmp
innobackupex: Waiting for ibbackup (pid=15959) to suspend
innobackupex: Suspend file '/root/db/2014-07-22_11-52-22/xtrabackup_suspended'
xtrabackup_51 version 2.0.8 for MySQL server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 3
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
>> log scanned up to (94 3209639289)
140722 11:52:25 InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: Error: could not open single-table tablespace file
InnoDB: ./roei_cc/transport.ibd!
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.
[root@ads-dev1 ~]#
I've checked permissions on the files but couldn't find a problem, I also tried running the command using mysql user and got the same error and even ran the next command:
# ulimit -n 4096 ; innobackupex /root/db
But got the exact same error, any ideas?
From doing a little research on error code 24, this seems to be a limit in place on the number files any one user can have open.
You can find these limits by running ulimit -Hn
and ulimit -Sn
. the -Hn is the unpassable limit, set by root.
Referencing http://www.percona.com/forums/questions-discussions/percona-xtrabackup/8243-innobackupex-1-5-1-error-in-backup
You will have to up the limit of open files until the error goes away. As root user, first: ulimit -a
to find the current maximum number of open files, then set a new limit with ulimit -n 2048
/ ulimit -n 4094
etc, probably incrementing to the power of 2 until you find the right size for your implementation.