Skip to content
ToolPlankDEV
Text Tools
100% In-Browser Execution

Updated:

Remove Duplicate Lines Online

Paste a list and remove duplicate lines online in one step. The first copy of each line stays in its original place. You can ignore case, trim spaces, sort the result, or list only the duplicates. Nothing is uploaded.

Options:

Your list

Input
0 chars · 0 lines

List without duplicates

Ready
Processing in browser...
0 chars · 0 linesUpdates as you type

Processed on your device. Your text is not sent to our servers.

More free tools you may need next.

Quick Guide

How to use Remove Duplicate Lines

Simple step-by-step instructions to get the exact output you need.

  1. 01

    Paste your list

    Put one item on each line in the Your list box, or press Paste to read the clipboard, or drop a .txt file onto the box. Press Example to load a short list with repeated lines.

  2. 02

    Choose how lines are compared

    By default two lines match only when they are exactly the same. Untick Match case to treat Apple and apple as one line. Tick Trim spaces around each line when stray spaces at the edges make equal lines look different.

  3. 03

    Pick what to keep

    The first copy of every line stays where it was. You can also sort the result, remove blank lines, show only the lines that have duplicates, or show only the lines that appear once.

  4. 04

    Copy or download the result

    The list without duplicates updates as you type. The line counts above each box show how many lines you started with and how many are left. Press Copy, or Download to save a .txt file.

How to Remove Duplicate Lines Online

To remove duplicate lines online, paste the text with one item per line and read the result on the right. The tool walks down the list once. The first time it sees a line it keeps it. Every later line that matches is dropped. The order of what is left is the order you pasted.

Here is the Example list with the default settings. Match case is on, so Apple and apple count as different lines, and the blank line is treated as a line like any other:

Text
Input             Output
apple             apple
banana            banana
Apple             Apple
cherry            cherry
banana            (empty line)
(empty line)
cherry
(empty line)
apple

Nine lines went in and five came out. The second banana, the second cherry, the second and third blank lines and the last apple were duplicates of lines that appeared earlier.

Delete Duplicate Lines Without Changing the Order

Sorting a list is a common way to find repeats, because equal lines end up next to each other. The cost is that your order is gone. This tool does not sort unless you ask it to. It keeps the first copy of each line in its original place, so a ranked list, a log in time order or a playlist still reads the way it did.

If you do want an alphabetical list, tick Sort A to Z. The sort is natural, so item2 comes before item10, and it is applied after the duplicates are removed. Sort Z to A gives the reverse. If both boxes are ticked, Z to A is used.

Match Case, Trim Spaces and Remove Empty Lines

Most lists that look like they have no duplicates do have them, hidden behind small differences. Three options deal with the common ones.

  • Match case: when it is off, Apple, apple and APPLE count as the same line. The tool keeps the first spelling it saw.
  • Trim spaces around each line: removes spaces and tabs at the start and end of every line before comparing. The trimmed text is what appears in the result.
  • Remove empty lines: drops blank lines and lines that hold only spaces. Without it, one blank line is kept because a blank line is a line too.

With Match case off, Remove empty lines on and Sort A to Z on, the same Example list becomes this:

Text
apple
banana
cherry

Spaces inside a line are never touched. Two lines that differ in the middle, such as new york and new york, stay separate. Some repeats are caused by characters you cannot see, such as a zero width space. The Invisible Character Remover shows those, and you can then run the cleaned text through this tool.

Find Duplicate Lines Instead of Deleting Them

Sometimes you want to know which lines repeat, not to throw them away. Tick Show only duplicated lines to get every line that appears more than once, listed one time each. With Match case off, the Example list gives:

Text
apple
banana
cherry
(empty line)

Show only lines that appear once does the opposite. It drops every line that has a duplicate, including its first copy, and keeps what is left. With default settings the Example list leaves only Apple, because it is the only line that appears exactly once. The two options cannot be combined, and the tool tells you if both are ticked.

Common Uses for a Duplicate Line Remover

  • Email, URL and keyword lists that were merged from several sources.
  • Log files and exported lists where the same entry was written many times.
  • Lists of names, IDs, product codes or tags pasted from different places.
  • Configuration and allow lists where a repeated entry causes a warning. If the cleaned list has to go inside a JSON string afterwards, the JSON Escape tool handles the quotes and line breaks.
  • Cleaning a column copied from a spreadsheet. If you are working in a spreadsheet file, see the Excel Case Converter for fixing the case of a whole column first.

A spreadsheet has its own command for this: in Excel, select the column and use Data, then Remove Duplicates. That works on whole rows or chosen columns. This tool is for plain text, where each line is one value. Once the repeats are gone, Sort Lines puts the list in order, Add Text to Each Line wraps every item in quotes or tags, and Column to Comma Separated List turns the list into one line. To see which items two lists share and which only one has, use Compare Two Lists.

Remove Duplicate Lines in Code and on the Command Line

If you do this often inside a script, these one-liners do the same job. The first keeps the original order, like this tool. The second sorts the lines and so loses the order.

Bash
awk '!seen[$0]++' list.txt > unique.txt
sort -u list.txt > unique.txt
Python
lines = open("list.txt", encoding="utf-8").read().splitlines()
unique = list(dict.fromkeys(lines))
print("\n".join(unique))

In Python, a dictionary remembers insertion order, so dict.fromkeys drops repeats and keeps the first copy of each line.

Limits and Things to Know

  • Lines are compared exactly, character for character. A letter with an accent and the same letter without it are different lines, and so are two visually identical letters that are built from different Unicode characters.
  • Case matching is done with the browser's standard lowercase rule. It is not a language-aware comparison.
  • Windows line endings are converted to plain line breaks when text is loaded, so the result uses one kind of line break throughout.
  • A trailing line break at the end of the text counts as a blank line. Tick Remove empty lines if you do not want it.
  • The tool does not detect near duplicates such as lines with a typo. Only identical lines, after your chosen options, are removed.
  • Input is limited to 2,000,000 characters. Very long lists may take a moment to show.

Your List Stays in Your Browser

The lines are compared by code running in your browser. They are not sent to a server and are not stored. See the Privacy Policy for details. Closing or reloading the page clears the text.

Frequently Asked Questions

Technical Q&A for Remove Duplicate Lines

Common questions about edge cases, character encoding, and privacy guarantees.

Paste your list into the box, one item per line. The result on the right shows each line once, keeping the first copy. Press Copy or Download to use it.

Leave both sort options off, which is the default. The first copy of every line stays where it was and only the later copies are removed.

Untick Match case. Apple, apple and APPLE then count as one line, and the tool keeps the first spelling it finds.

Yes. Tick Trim spaces around each line. Lines that differ only by spaces or tabs at the start or end are then treated as the same line.

Tick Show only duplicated lines. The result lists every line that appears more than once, one time each.

Tick Show only lines that appear once. Any line that has a duplicate is dropped completely, including its first copy.

Only if you choose to. Sort A to Z and Sort Z to A are optional and use natural order, so item2 comes before item10. Without them the original order is kept.

Paste the list into the box, one item per line. Each line is kept once, and you can ignore case, trim spaces or sort the result with the options above the box.

They are not exactly identical. The usual causes are different capital letters, spaces at the ends of a line, or invisible characters. Try Match case off and Trim spaces on, or check the text with the Invisible Character Remover.

No. The comparison runs in your browser and the text is not sent to a server or saved. Reloading the page clears it.