Unix sort for partially ordered data sets

It would be easier to split the file into smaller sections and sort those. To split:-

split --lines=100000 large_file file_part.

Then sort each of those by using normal sort

for suffix in `ls file_part.* | cut -f2 -d.` 
do 
  sort file_part.${suffix} > file_sorted.${suffix} 
done

you can then combine by merge sorting

sort -m file_sorted.*

That should be much easier on your machine.

Related