Why do file transfers between drives use RAM

I've noticed that whenever I copy or move large files from my SSD, which I use as the system drive, to my HDD or to an external hard or flash drive, the speed graph shown by Windows always looks the same: the transfer speed starts at around 450 MB/s, and after a few seconds drops down to somewhere between 90 and 130 MB/s and remains stable until the end of the copy/move operation.

This sparked my curiosity, so I decided to figure out what is the cause of this. Some of my thoughts were these:

Maybe that is the actual speed at which the transfer happens

Doubtfully. While the 450 MB/s speed matches the rated speed of my SSD, considering I also have some other disk reads/writes going on in the background, there is no way a 7200 rpm hard drive is capable of keeping up with it, as the 130 MB/s speed I get later on is also the most I can expect from it. So, where does the extra data go?

The extra data is being stored in the hard drive's cache memory

This makes a bit more sense, but if I take into account the duration of the higher transfer speed, my hard drive's cache would have to be over 3 GB in size, which it definitely isn't. What else could it be?

The extra data is being stored in the RAM

This makes sense. My RAM is the only other part of my system which can match the speed of my SSD, and I have plenty of it. Let's check this theory!

I open up the Task Manager, and take a look at the Performance tab. Memory usage is stable at 3.7 GB. Then I start another 15 GB file transfer. Memory usage starts rising, and stops at 5.3 GB just as the transfer speed drops to 130 MB/s. It remains the same until the end of the file transfer(the transfer dialog closes), and then slowly drops back to the 3.7 GB level it was before the transfer.

So, my last theory is true. Further confirmation is the fact that the extra used memory is marked as Modified

.

What's the point?

My question is, what is the purpose of doing this? While I don't mind having some of my RAM used by file transfers, as even during the heaviest of my multitasking sessions I've never seen its usage go over 70%, what is the benefit of storing 1.6 GB of data which you won't be doing any kind of processing on in your RAM?

I don't see any benefit from the data integrity standpoint, as you're merely copying the files, and in the case of a power failure neither the RAM or the HDD will be particularly successful in retaining the data in transfer.

I could see the benefit being that the source disk(the SSD) is quickly freed up, so that if another process needs to perform lots of read/write operations on it, it can do so without the file transfer impeding it, but if that is the case, why not go ahead and load all 15 GB at max speed into the memory?

Also, this process misleads the user, as the file transfer keeps going even after the transfer dialog closes, because some of the data is still being copied from the memory to the hard drive. This could cause a user to plug out a removable drive while data is still being written to it, possibly corrupting the removable drive, cause not everyone bothers with safely removing hardware.

Keep in mind I haven't thoroughly tested this with removable drives, as Windows might be handling them differently, making my last point invalid.

Solution 1:

Why do file transfers between drives use RAM?

Because I/O operations are (almost always) between a peripheral and RAM.
So a file copy is really two disk operations: a read (to RAM) and then a write (from RAM).

Some systems can perform peripheral-to-peripheral operations (and therefore not require a buffer in RAM). I've seen SCSI host adapters that can perform drive-to-drive transfers (without the involvement of CPU and RAM by using an on-board processor and RAM/FIFO). I've seen DMA controllers that can perform peripheral-to-peripheral transfers. These are the exceptions, not the rule nor in common use.

Note that error handling is doubly complicated when using peripheral-to-peripheral transfers, and is therefore rarely used by an OS for arbitrary I/O operations even when the hardware is available.

My question is, what is the purpose of doing this?

The source data (the file being read/copied) has to be read into RAM simply because that is the nature of computer archtecture (combined with the use of DMA).
Back in the day when RAM was scarce and OSes were not as sophisticated, the typical choice was to perform these transfers with a buffer as small as possible (e.g. just one sector or block), or a buffer that would optimize speed (e.g. a multi-sector transfer).
Nowadays with relatively large memory and sophisticated algorithms, the typical OS will try to use any/all of free/unused memory for the buffer to read this source data.

After that source data has been written to the destination, that data can be discarded. But if there is no demand for this memory (i.e. it will remain unused), then there is no imperative to discard this data.
And if the data has been tagged or cataloged to identify the file it came from, then this data can be cached for possible reuse.

Note that this cache of files is essentially free; there was no burden of additional I/O on the OS to obtain this cache. The only cost is maintaining the catalog of the contents, which can be offset when there's a cache hit (and save the I/O of a reread).

So the caching of files is a free consequence of normal reading of those files. It temporarily uses memory that would otherwise be idle and unused. The overhead in the OS for maintaining this cache is typically minimal and is paid back when cache hits occur.

Also, this process misleads the user, as the file transfer keeps going even after the transfer dialog closes, because some of the data is still being copied from the memory to the hard drive.

That is a trade-off between system availability to the user versus guaranteed writes.
A program can be written to perform blocked synchronous writes, which would wait for each physical write operation to complete before the program continued. Or the program, at judicious points, could wait for the physical write operations to complete by invoking a sync() or similar system call.

In a write (or copy) operation, the modern OS will try to be available to the user as soon as all write operation have been at least queued up (meaning that they may not have completed).
This facilitates multitasking. If you wanted to do something else that didn't even involve that drive, why should you have to wait for operations on that drive to finish to do that something else?
The trade-off is that you have to be an educated user, and be cognizant that (storage) devices need to be properly unmounted.

This availability feature has nothing to do with the above mentioned file cache feature. If it were available/possible, you could disable one without affecting the other.

Note
A data transfer could use no RAM buffer if:
1. programmed I/O (using the CPU) is performed for both the input and the output operations,
AND
2. the input and output have matching data transfer rates and transfer sizes,
AND
3. both devices are character-oriented and not block devices. (This would exclude disk drives.)

However an OS will rarely choose to use programmed I/O when DMA or bus mastering is available, nor is likely to be programmed to handle the double complexity of a paired input+output transfer.