Moving large amount of files (~ 100 000)

I work with folders that contain a lot of files, like 100 000 or even 1 000 000 files per folder. When I try to move the content of a folder into another, my computer always gets stuck. Even when the process seems finished, I can't see the content of any folder because nautilus seems completely frozen and I have to force my computer to restart. I noticed that this happens also when I try to move like 10 000 files.

Is that a problem of my computer or is it normal when working with these numbers?

Any smart way of performing this file transfer?

Solution 1:

Perhaps consider using a pure command line method to transfer very large amounts files, you will undoubtedly find the process is substantially faster than using a gui.

There are many different ways to accomplish this, but the following worked quickly, safely and efficiently on my system:

find . -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Some explanation for this command:

1. Your input directory is the '.' character and for this particular command you need to be in that directory
2. Your output directory is the <destination> in my example. Obviously modify this to suit your own needs and leave out the brackets.
3. This syntax allows for filenames with spaces as a bonus :)

Endless permutations are possible but this should work well and much more efficiently than the gui. One permutation for example: if you wanted to move only pdf files you could run:

find . -iname "*.pdf" -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Use of xargs opens many possibilities particularly with the movement of such a large number of files. Many, many possibilities....

Potential Problems:

There are at least 2 potential pitfalls to ponder, thanks to the commenters below for these thoughts:

1. Your destination directory could be corrupt, in a subsequently unreachable location, mistyped etc. mv will still move the files there! Be careful here...
2. If the -t option (--target-directory) is missing and the destination folder is actually a file you will move one file and fail on the rest. mv has 2 uses: rename source to destination or move source to directory. Again be careful...

Solution 2:

I had similar experience before, it is normal when dealing with a large number of files. I was having a large collection of PDF data-sheets (electronic parts).

GUI tools check for some file details & meta data (Icon/Thumbnail, Size, ...), it will be a big deal in such case. Even in Icon View and without thumbnails, they will freeze as most of them are not designed for such extreme case. GUI tool try to load presentation icons for all files/folders in directory even those items are not visible to user in current screen portion. Sorting is also part of the problem and no way to avoid it.

• I end up splitting files files on separate folders based on brand/model less then 10000 each. May be you can use date (as most people do with photos/scans) or first letter(s) (as in Ubuntu package repository)
• It is easier to use CLI tools instead as they show only what you have requested. You can use locate for quick search in stead of find.

• For move operation, use mv in terminal (GUI tools are slow because they try to update the view periodically).

  If it is in the same partition, the command will change only the pointers in the File-system index. If not, then it will be dual operation (copy & delete). That will be expensive.

There is only one case I can help, If you are copying those files multiple times and they are not updated. As I did when I share my collection with friends, each time I try to copy it takes a decade. (This is more useful with small size files only)

• Create a single package or few packages, like zip with no/low compression. When you copy it, it will faster, so let DMA do its job.