Remove duplicate lines

How to use it#

1. Paste your list into the box above, one entry per line.
2. Choose whether to keep one of each or drop every repeated line entirely.
3. Turn on ignore case or trim spaces if near-matches should count as the same line.
4. Copy the result, or download it as a text file.

The order is kept, which matters more than it sounds#

Plenty of ways to deduplicate a list sort it on the way through, because sorting is how the duplicates get found. That is fine for a set of tags and destructive for anything where the order carries meaning: a log, a playlist, a ranked shortlist, a set of steps.

This keeps the first time each line appeared and leaves everything where it was. If you do want the result sorted, the line sorter has a remove duplicates option and does both in one pass.

When two lines are nearly the same#

Exact matching is the default because it is the only one that cannot surprise you. It also misses the duplicates people actually have. A list pasted out of a spreadsheet arrives with trailing spaces on half the rows, and a list of email addresses collected from a form arrives with the capitalisation the typist happened to use.

Trim spaces around each line handles the first, and it trims the output too rather than quietly matching one thing and giving you another. Ignore case handles the second, and the spelling you keep is the first one that appeared. Both settings apply as soon as you tick them, so you can see how many more lines go.

Keeping one of each, or dropping repeats entirely#

Keep one of each is the usual meaning of deduplicating: every distinct line survives once. Drop all repeats is the stricter reading, where a line that appeared more than once is removed completely, first copy included.

The second is what you want when the repeats are the problem rather than the surplus. Finding which order numbers were submitted only once, or which entries in a merged list came from a single source, is the same question asked of a list of lines.

What counts as a line#

A line ends wherever your text does: a Unix newline, a Windows carriage return and newline, or an old Mac carriage return. All three are read, and the result comes back with the ending your text arrived with, so a file pasted from Windows does not come back reformatted.

Blank lines are lines. Two of them are duplicates of each other, so a list padded out with empty rows collapses to one unless you tick remove blank lines, which drops them all.