how to merge large txt files of 10GB+ into 1 txt file and remove duplicates lines from this 1 txt file fastest way? [duplicate]
Solution 1:
If you are using Linux, you could do it like that:
cat aa.txt bb.txt | sort -u > newfile.txt
Here aa.txt is the first text file and bb.txt the second one.
sort -u sorts the file alphabetically and with -u (see also here https://stackoverflow.com/a/9377125/7311363) you're eliminating duplicates. With > newfile.txt you're writing that to newfile.txt.