How to Remove Duplicate Lines
Duplicate lines creep into lists whenever you combine exports, copy from several sources or append logs. Here’s how to remove them quickly — and how to decide which copy to keep.
Open Remove Duplicate LinesStep by step
Open Remove Duplicate Lines
Go to the Remove Duplicate Lines tool. It works on any device and needs no account.
Paste your text or upload a file
Paste a list with one item per line, or drag in a .txt, .csv, .docx or .pdf file. Files are read on your device.
Choose how lines are matched
Turn off “Case-sensitive” if “Apple” and “apple” should count as the same. Keep “Trim whitespace” on so stray spaces don’t create false differences.
Remove the duplicates
Press Remove duplicates. The result keeps the first occurrence of every line, in the original order, and tells you how many lines were removed.
Copy or download
Copy the cleaned list, or download it as TXT, CSV, JSON, Word or PDF.
Example
Apple apple Banana Apple Orange Banana
Apple Banana Orange
With case-sensitive matching turned off.
First or last occurrence?
Keeping the first occurrence preserves the order in which items first appeared — usually what you want for lists. Keep the last occurrence when later lines are more recent, such as the newest entry for each ID in a log export.
In Excel
Select the column, then choose Data → Remove Duplicates. Excel compares cells without regard to case and deletes the duplicate rows in place, so work on a copy if you need the original.
In Google Sheets
Use the UNIQUE function in an empty column — it returns each distinct value once and updates automatically when the source changes.
=UNIQUE(A2:A)On the command line
On macOS or Linux, awk removes duplicates while keeping the original order; sort -u is shorter but sorts the result.
awk '!seen[$0]++' list.txt > unique.txtTry Remove Duplicate Lines
Delete repeated lines and keep one copy of each.
Frequently asked questions
How do I see which lines were duplicated?
Use Find Duplicates. It lists each repeated line once, with how many times it appears.
Can I remove duplicates across two separate lists?
Yes — use Merge Lists with “Remove duplicates” on, or Compare Two Lists to see what they share.
How large can my list be?
Hundreds of thousands of lines. Large inputs are processed in a background thread so the page stays responsive.