Does downloading/copying files simultaneously promote fragmentation?

Does writing more than one file simultaneously to a hard drive partition promote fragmentation? Would these files be written interlaced on the hard drive?

Examples:

• Downloading several files simultaneously from the web
• Copying files from various sources to the same partition
• Downloading several torrents

Is this different between various OSs? Is allocating space a priori for the entire file an application specific feature?

Solution 1:

It depends a lot on the filesystem in use, to some extent operating system, as well as the program that's opening the file. It is up to the application to request the correct amount of storage when creating a new file, and it is the operating system and file system features that affect the open-and-extend behavior these apps follow when they don't know how much space to ask for.

If a program asks the operating system for a 4.2GB file, the OS will find a hunk of space with 4.2GB free. Failing that it'll give the program enough freespace fragments to save the file. There are some variables here, this is simplified.

If a program asks the operating system for a 4KB file, the OS will give it. If the program keeps asking for new contiguous 4KB segments (you're downloading a file, and the program didn't know how big it was) until it gets 4.2GB of space, the OS will happily give space. This is the case where filesystem and OS make differences.

XFS, for instance, is extent-based so added space will be contiguous until it can't be. Also, XFS can be configured to pre-allocate new files with a certain amount of space which can further minimize fragmentation (useful on Virtual Machine storage volumes, as well as DVD ISO volumes).

EXT2/3, NTFS, and I believe Reiser, will allocate contiguous until it can't anymore. Then it'll start new fragments. Each fragment could be 4KB in size though.

Lets take the worst-case. Two parallel 4.2GB files being saved 4KB at a time in parallel, to the same directory. The 'same directory' thing is part of the worst-case since some OS/FS combinations attempt to keep files in the same directory near each other. For some OS/FS combinations this will result in the two files being completely interleaved on the disk. For others that try to keep new-open files on different free segments there may be no fragmentation at all or very little.

A bit on XFS (I think ext4 and btrfs also can do this). In this case, the interleave will not be 4KB, it'll be whatever you set your allocation-size value to when mounting the volume. Set it to 256MB, and you'll get a 4.2GB file with 17 fragments instead of over a million. Useful for volumes storing large files like media servers. Not useful for volumes storing lots of itty bitty files and a few big ones.

It is because of this that Bittorrent clients either open sparse files when starting a new download or just allocate the specified size in total before starting.

When copying files from different parts of the same partition (or heck, even from other partitions) the copy process theoretically knows how big the copied files are so requests space of the appropriate size from the OS from the start; no need to append allocations for new files.