How to only remove duplicate lines if they're immediately after each other in the file
Lets say I have the following file:
$ cat test.txt
a
-----
b
-----
-----
c
-----
-----
-----
d
-----
e
-----
-----
Now I want to remove all the -----
, but only if they're repeating after each other. So the result should look like this:
a
-----
b
-----
c
-----
d
-----
e
-----
I tried grep -Pvz -- "-----\n-----"
, but that didn't work.
Solution 1:
That's exactly what the uniq
command is made for:
NAME
uniq - report or omit repeated lines
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT
(or standard output).
With no options, matching lines are merged to the first occurrence.
So
$ uniq test.txt
a
-----
b
-----
c
-----
d
-----
e
-----
Alternatively, you can use this sed one-liner 69. Delete duplicate, consecutive lines from a file (emulates "uniq") from Sed One-Liners Explained, Part III: Selective Deletion of Certain Lines and Special Applications
sed '$!N; /^\(.*\)\n\1$/!P; D' test.txt
which might be preferred if you want to edit test.txt
in place (by adding the -i
or --in-place
option).