Removing duplicate rows in Notepad++

Is it possible to remove duplicated rows in Notepad++, leaving only a single occurrence of a line?

Solution 1:

Notepad++ with the TextFX plugin can do this, provided you wanted to sort by line, and remove the duplicate lines at the same time.

To install the TextFX in the latest release of Notepad++ you need to download it from here: https://sourceforge.net/projects/npp-plugins/files/TextFX

The TextFX plugin used to be included in older versions of Notepad++, or be possible to add from the menu by going to Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX -> Install. In some cases it may also be called TextFX Characters, but this is the same thing.

The check boxes and buttons required will now appear in the menu under: TextFX -> TextFX Tools.

Make sure "sort outputs only unique..." is checked. Next, select a block of text (Ctrl+A to select the entire document). Finally, click "sort lines case sensitive" or "sort lines case insensitive"

Solution 2:

Since Notepad++ Version 6 you can use this regex in the search and replace dialogue:

^(.*?)$\s+?^(?=.*^\1$)

and replace with nothing. This leaves from all duplicate rows the last occurrence in the file.

No sorting is needed for that and the duplicate rows can be anywhere in the file!

You need to check the options "Regular expression" and ". matches newline":

• ^ matches the start of the line.

• (.*?) matches any characters 0 or more times, but as few as possible (It matches exactly on row, this is needed because of the ". matches newline" option). The matched row is stored, because of the brackets around and accessible using \1

• $ matches the end of the line.

• \s+?^ this part matches all whitespace characters (newlines!) till the start of the next row ==> This removes the newlines after the matched row, so that no empty row is there after the replacement.

• (?=.*^\1$) this is a positive lookahead assertion. This is the important part in this regex, a row is only matched (and removed), when there is exactly the same row following somewhere else in the file.

Solution 3:

If the rows are immediately after each other then you can use a regex replace:

Search Pattern: ^(.*\r?\n)(\1)+

Replace with: \1

Solution 4:

If you don't care about row order (which I don't think you do), then you can use a Linux/FreeBSD/Mac OS X/Cygwin box and do:

$ cat yourfile | sort | uniq > yourfile_nodups

Then open the file again in Notepad++.