What is the difference between keeping first and last occurrences?

First occurrence keeps the earliest version of each line in your list and skips every later repeat, preserving the original order. Last occurrence overwrites the earlier entry with the most recent version of that line, which is useful when later rows in a list — such as updated statuses or newer metadata — should win over earlier ones.

Does the remover preserve the original line order?

Yes. Lines are processed top to bottom and only exact repeats are dropped, so the surviving lines stay in their original relative order. This is deliberate: alphabetical sorting would reorder your data, but a dedupe tool should only remove redundant entries and leave everything else untouched.

What if my lines have different amounts of whitespace?

Enable Trim before compare to ignore leading and trailing whitespace, so "apple" and " apple " are treated as the same line. The output keeps the text of whichever occurrence survives according to your Keep setting, so with Keep set to First occurrence the untrimmed earlier version is the one that appears in the result.

How do I know how many duplicates were removed?

The tool shows three statistics — Input, Output, and Removed — plus a badge summary in the "in → unique" format that reports the total duplicates removed. The Removed stat is highlighted when it is greater than zero, making it easy to confirm the cleanup worked at a glance.

Is there a limit to the list size?

There is no hard limit beyond your browser memory, and the deduplication runs in a single pass using a hash map, so even lists with thousands of lines are processed in milliseconds. The result is computed locally and never uploaded, which also makes the tool safe for private data like email lists or internal logs.

Text

Duplicate Line Remover

Remove duplicate lines, optionally case-insensitive.

duplicatelinesuniquededupe

Loading tool…

About the Duplicate Line Remover

Duplicate lines are the classic symptom of merged lists: combined email reports, concatenated logs, union queries that went wrong, or copy-paste accidents. They waste space, skew counts, and make deduplication in a spreadsheet feel harder than it should. The Duplicate Line Remover tool cleans a list in one pass, preserving the original order of the lines it keeps. Paste your list into the Lines area — one entry per line — then fine-tune the comparison with two switches: Case-insensitive makes Apple and APPLE match, and Trim before compare ignores stray leading or trailing whitespace. The Keep radio group decides which occurrence survives: First occurrence keeps the earliest line and drops later repeats, while Last occurrence overwrites with the final version of each line, which is handy when newer entries supersede older ones. Click Remove duplicates and the output appears in the Unique lines box along with Input / Output / Removed statistics and a badge summary. Whether you are cleaning a mailing list, a playlist, or a manifest, this dedupe tool gives you repeatable, auditable results instantly.

Hand-written guide

Examples

Input

apple
banana
apple
cherry

Output

apple
banana
cherry

Note: Default settings: keep the first occurrence, case-sensitive. 4 lines in, 3 unique.

Input

Apple
apple
Banana

Output

Apple
Banana

Note: With Case-insensitive on, Apple and apple match and only the first variant survives.

Input

Apple
apple

Output

apple

Note: Case-insensitive with Keep set to Last occurrence — the newest variant overwrites the older one.

How to use

1
Paste your list into the Lines area, one entry per line.
2
Toggle Case-insensitive to make Apple and APPLE match.
3
Toggle Trim before compare to ignore leading and trailing whitespace.
4
Choose First occurrence or Last occurrence in the Keep radio group.
5
Click Remove duplicates and review the Unique lines output plus the Input / Output / Removed stats.

Common use cases

Deduplicating merged mailing lists before a campaign send.
Cleaning concatenated server logs for analysis.
Removing repeated rows from exported spreadsheet data.
Consolidating tag or category lists exported from a CMS.
Preparing unique inventory SKUs from multiple source files.
Cleaning up union queries in SQL results pasted for review.

Best practices

Use default exact matching for structured data where casing is deliberate, like SKUs or file paths.
Enable Case-insensitive plus Trim before compare for human-entered lists with inconsistent formatting.
Choose Keep based on semantics: first for stable identifiers, last for values that evolve over time.
Note that order is preserved by design — run a separate sort if a reordered list is required.
Cross-check the Removed stat against your expected duplicate count as a sanity check.

Tips

Sort is not part of dedupe: the output keeps original order, so run a sorter afterwards if you want an ordered list.
Use Keep set to Last occurrence when rows carry version-like values and the newest entry should win.
Toggle Case-insensitive for user-entered data like email lists, where capitalization is inconsistent.
The Removed stat is highlighted when greater than zero, giving you instant confirmation the dedupe ran.

Frequently asked questions

Two lines are duplicates when their comparison keys match, where the key is the line after optional trimming and case folding depending on your switches. With default settings the comparison is exact and case-sensitive, so Apple and apple are different; enabling Case-insensitive makes them equal, and Trim before compare ignores surrounding whitespace.

Explore more text tools

Browse the full collection of text tools on the hub, or jump back to all categories.

All Text tools All categories