Insert sectors into a file efficiently

I have a large file (multiple gigabytes) and I need to repeatedly insert sectors in the beginning of it (the exact number of sectors and writes isn't known ahead of time).

One obvious solution is to simply read the file, insert the data and write again, but that's far too inefficient. Considering the way how the filesystem works, it's definitely possible to insert a sector to any place of the file, but is this possible without digging very deeply into very low level APIs?

Solution 1:

I doubt you are going to find an API that lets you do this. You could perhaps look at the disk defragmentation API but I'm not sure if it gives you enough control.

If you can require NTFS I have an alternative suggestion; sparse files. Create a really large sparse file (1 TiB?) and write to the file from end to start. You would probably need some kind of header at the start of the file that notes the offset where you actually are in the file. This scheme requires that you know the size of each chunk before you start writing a chunk. And obviously, this will not work if you are just writing a plain ASCII log file.

A major drawback of this design is that it becomes hard for a user to move/copy the file to another volume.