How to remove duplicate lines on Mac

You have a list — email addresses, log lines, a .gitignore, a bibliography — and the same entry appears more than once. You want each line kept exactly once.

The Terminal way

macOS ships with the tools, but they carry a catch. The obvious command reorders your file:

sort -u input.txt > output.txt

sort -u deduplicates by sorting first, so the surviving lines come out in alphabetical order. If the order of your list carried meaning — a sequence of steps, a chronological log, a hand-curated list — it is gone.

uniq has the opposite problem: it only collapses runs of adjacent identical lines, so it does nothing at all unless the file is already sorted.

To deduplicate and keep the original order, you need awk:

awk '!seen[$0]++' input.txt > output.txt

That works. It also means remembering an idiom, redirecting to a second file because you cannot safely write to the file you are reading, and then moving that file back over the original.

The KlarText way

Open the file, then:

Edit ▸ Text Transforms ▸ Remove Duplicate Lines

The first occurrence of every line stays exactly where it was. Later copies are removed. Nothing is reordered.

If text is selected, the transform is applied to that selection, expanded to whole lines — so you can clean up one block of a file and leave the rest untouched. With nothing selected it applies to the entire document. Either way it is one undo step: ⌘Z puts the file back.

Details worth knowing

Matching is exact. Two lines count as duplicates only if they are identical character for character. [email protected] and [email protected] — with a trailing space — are two different lines and both survive. If a deduplication leaves more behind than you expected, this is almost always why. Run Edit ▸ Text Transforms ▸ Strip Trailing Whitespace first, then deduplicate. See how to remove trailing whitespace on Mac.

Case matters too. Alice and alice are distinct lines and both are kept.

To get sort -u behaviour, run Sort Lines (A→Z) and then Remove Duplicate Lines. Two menu items, still one file, still undoable. See how to sort lines in a text file.

Blank lines are lines. A document with many empty lines between paragraphs will keep the first blank line and drop the rest. Select just the block you mean if that is not what you want.