Removing duplicate rows in vi?
I have a text file that contains a long list of entries (one on each line). Some of these are duplicates, and I would like to know if it is possible (and if so, how) to remove any duplicates. I am interested in doing this from within vi/vim, if possible.
If you're OK with sorting your file, you can use:
:sort u
Try this:
:%s/^\(.*\)\(\n\1\)\+$/\1/
It searches for any line immediately followed by one or more copies of itself, and replaces it with a single copy.
Make a copy of your file though before you try it. It's untested.
From command line just do:
sort file | uniq > file.new