Is there a better way to transfer data from a non-Windows file system to Windows?
When designing our next-gen airborne LIDAR system, we intended to store our data on storage drives that were formatted with NTFS. This is the way our previous-gen systems work. Our systems run Linux (CentOS) but using NTFS hasn't caused us any issues. It also allows our customers to remove the storage drives and connect them directly to a Windows PC for processing.
As our system collects data, it is divided up and written to multiple SATA drives (we do plan to move to removable NVME drives once the motherboard manufactures catch up to the times). Two thirds of the way through development we encountered a massive problem. Although storing the data to NTFS worked great for our last-gen systems, CentOS couldn't write to NTFS fast enough to handle the increased data flow of our new system. The only solution we could come up with at the time was to reformat the storage drives to a native Linux file system. We chose ext4 and got the write speeds we needed.
Problem 1: Our customer now has several terabytes of data on ext4 drives while the processing software runs on a Windows PC.
Our Current Solution: We have the customers install "Linux Reader" software that allows them to copy the data from the ext4 drives.
Other Solutions We've Tried: Having the customers use ext2fs software. Also, having them boot the processing PCs into Ubuntu, transferring the data, then rebooting back into Windows.
Problem 2: The solutions for problem 1 feel unprofessional. They also each have their own set of problems. For example, the Linux Reader software sometimes doesn't see all of the files on the drives.
I've had a few additional ideas that I haven't tried yet (such as running Ubuntu on a VM on the processing PCs to read the drives and copy the data). Its hard to justify continuing down this rabbit hole when we already have a solution that "works" (mostly). It's just that when the customer pays such a large sum for professional equipment, they shouldn't have to deal with shoddy "workarounds" just to process their own data.
Does anyone here have any ideas or suggestions for a better solution?
Edit 1: I wanted to clarify that I'm not asking this question on behalf of my company. They are fine with the current solution for now. I personally want a better solution for the customers as I am unsatisfied with what we currently have.
If you really want to use a native Windows filesystem then currently there are only a few solutions:
- NTFS: Previously on Linux the FUSE ntfs3g driver is used. It runs in userspace so it's extremely slow compared to a driver in kernel space. The new Paragon driver has just been merged into Linux 5.15 so since that onward performance should be great. But since CentOS are always on ancient kernels you might need to ditch that for a better distro, or just avoid NTFS
- exFAT: It has a great kernel driver since Linux 5.7. But again you'll need to have a new kernel version which might not be possible with CentOS. There were some leaked kernel drivers for exFAT in the comment below that may be worth a try
- UDF: I don't know performance of the Linux UDF driver so I can't provide recommendations on this. But if you do wanna try this then beware that it's tricky to make a real universal UDF partition. See Can and should UDF be used as a hard drive format? for a script to do this
So IMO a Linux driver with great Windows support should be used. Currently there's no good ext4 driver on Windows so you should use Btrfs or ZFS instead. They're both more superior than ext4 with far better Windows drivers:
- For Btrfs use WinBtrfs. It's extremely reliable and is an official component of ReactOS, so probably that's why Btrfs has great Windows support
- For ZFS use ZFSin which is a Windows port of OpenZFS. I didn't use this as much but AFAIK it's not as stable as WinBtrfs
If the client uses Windows 10 then you can even mount any Linux filesystems on wsl2
Why not use exFAT? It's small, has huge capacity, Linux can write to it, there are no issues with user permissions and there is reduced overhead.