'Argument list too long' error while copying a large number of files

Solution 1:

*.jpg expands to a list longer than the shell can handle. Try this instead

find  /home/ftpuser/public_html/ftparea/ -name "*.jpg" -exec cp -uf "{}" /your/destination \;

Solution 2:

There is a maximum limit to how long an argument list can be for system commands -- this limit is distro-specific based on the value of MAX_ARG_PAGES when the kernel is compiled, and cannot be changed without recompiling the kernel.

Due to the way globbing is handled by the shell, this will affect most system commands when you use the same argument ("*.jpg"). Since the glob is processed by the shell first, and then sent to the command, the command:

cp -uf *.jpg /targetdir/

is essentially the same to the shell as if you wrote:

cp -uf 1.jpg 2.jpg ... n-1.jpg n.jpg /targetdir/

If you're dealing with a lot of jpegs, this can become unmanageable very quick. Depending on your naming convention and the number of files you actually have to process, you can run the cp command on a different subset of the directory at a time:

cp -uf /sourcedir/[a-m]*.jpg /targetdir/
cp -uf /sourcedir/[n-z]*.jpg /targetdir/

This could work, but exactly how effective it would be is based on how well you can break your file list up into convenient globbable blocks.

Globbable. I like that word.

Some commands, such as find and xargs, can handle large file lists without making painfully sized argument lists.

find /sourcedir/ -name '*.jpg' -exec cp -uf {} /targetdir/ \;

The -exec argument will run the remainder of the command line once for each file found by find, replacing the {} with each filename found. Since the cp command is only run on one file at a time, the argument list limit is not an issue.

This may be slow due to having to process each file individually. Using xargs could provide a more efficient solution:

find /sourcedir/ -name '*.jpg' -print0 | xargs -0 cp -uf -t /destdir/

xargs can take the full file list provided by find, and break it down into argument lists of manageable sizes and run cp on each of those sublists.

Of course, there's also the possibility of just recompiling your kernel, setting a larger value for MAX_ARG_PAGES. But recompiling a kernel is more work than I'm willing to explain in this answer.