Percona XtraBackup: Getting an error regarding "Too many open files", how can I solve this issue?

I'm trying to backup using Percona XtraBackup and getting the following error:

[root@ads-dev1 ~]# innobackupex /root/db

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013.  All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

140722 11:52:16  innobackupex: Starting mysql with options:  --unbuffered --
140722 11:52:16  innobackupex: Connected to database with mysql child process (pid=15930)
140722 11:52:22  innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql  Ver 14.12 Distrib 5.0.95, for redhat-linux-gnu (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /root/db/2014-07-22_11-52-22
140722 11:52:22  innobackupex: Starting mysql with options:  --unbuffered --
140722 11:52:22  innobackupex: Connected to database with mysql child process (pid=15953)
140722 11:52:24  innobackupex: Connection to database server closed

140722 11:52:24  innobackupex: Starting ibbackup with command: xtrabackup_51  --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/root/db/2014-07-22_11-52-22 --tmpdir=/tmp
innobackupex: Waiting for ibbackup (pid=15959) to suspend
innobackupex: Suspend file '/root/db/2014-07-22_11-52-22/xtrabackup_suspended'

xtrabackup_51 version 2.0.8 for MySQL server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 3
xtrabackup:   innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
>> log scanned up to (94 3209639289)
140722 11:52:25  InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: Error: could not open single-table tablespace file
InnoDB: ./roei_cc/transport.ibd!
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.
[root@ads-dev1 ~]# 

I've checked permissions on the files but couldn't find a problem, I also tried running the command using mysql user and got the same error and even ran the next command:

# ulimit -n 4096 ; innobackupex /root/db

But got the exact same error, any ideas?


From doing a little research on error code 24, this seems to be a limit in place on the number files any one user can have open.

You can find these limits by running ulimit -Hn and ulimit -Sn. the -Hn is the unpassable limit, set by root.

Referencing http://www.percona.com/forums/questions-discussions/percona-xtrabackup/8243-innobackupex-1-5-1-error-in-backup

You will have to up the limit of open files until the error goes away. As root user, first: ulimit -a to find the current maximum number of open files, then set a new limit with ulimit -n 2048 / ulimit -n 4094 etc, probably incrementing to the power of 2 until you find the right size for your implementation.