How to Remove Duplicate Lines Online
To remove duplicate lines online, paste the text with one item per line and read the result on the right. The tool walks down the list once. The first time it sees a line it keeps it. Every later line that matches is dropped. The order of what is left is the order you pasted.
Here is the Example list with the default settings. Match case is on, so Apple and apple count as different lines, and the blank line is treated as a line like any other:
Input Output
apple apple
banana banana
Apple Apple
cherry cherry
banana (empty line)
(empty line)
cherry
(empty line)
appleNine lines went in and five came out. The second banana, the second cherry, the second and third blank lines and the last apple were duplicates of lines that appeared earlier.
Delete Duplicate Lines Without Changing the Order
Sorting a list is a common way to find repeats, because equal lines end up next to each other. The cost is that your order is gone. This tool does not sort unless you ask it to. It keeps the first copy of each line in its original place, so a ranked list, a log in time order or a playlist still reads the way it did.
If you do want an alphabetical list, tick Sort A to Z. The sort is natural, so item2 comes before item10, and it is applied after the duplicates are removed. Sort Z to A gives the reverse. If both boxes are ticked, Z to A is used.
Match Case, Trim Spaces and Remove Empty Lines
Most lists that look like they have no duplicates do have them, hidden behind small differences. Three options deal with the common ones.
- Match case: when it is off, Apple, apple and APPLE count as the same line. The tool keeps the first spelling it saw.
- Trim spaces around each line: removes spaces and tabs at the start and end of every line before comparing. The trimmed text is what appears in the result.
- Remove empty lines: drops blank lines and lines that hold only spaces. Without it, one blank line is kept because a blank line is a line too.
With Match case off, Remove empty lines on and Sort A to Z on, the same Example list becomes this:
apple
banana
cherrySpaces inside a line are never touched. Two lines that differ in the middle, such as new york and new york, stay separate. Some repeats are caused by characters you cannot see, such as a zero width space. The Invisible Character Remover shows those, and you can then run the cleaned text through this tool.
Find Duplicate Lines Instead of Deleting Them
Sometimes you want to know which lines repeat, not to throw them away. Tick Show only duplicated lines to get every line that appears more than once, listed one time each. With Match case off, the Example list gives:
apple
banana
cherry
(empty line)Show only lines that appear once does the opposite. It drops every line that has a duplicate, including its first copy, and keeps what is left. With default settings the Example list leaves only Apple, because it is the only line that appears exactly once. The two options cannot be combined, and the tool tells you if both are ticked.
Common Uses for a Duplicate Line Remover
- Email, URL and keyword lists that were merged from several sources.
- Log files and exported lists where the same entry was written many times.
- Lists of names, IDs, product codes or tags pasted from different places.
- Configuration and allow lists where a repeated entry causes a warning. If the cleaned list has to go inside a JSON string afterwards, the JSON Escape tool handles the quotes and line breaks.
- Cleaning a column copied from a spreadsheet. If you are working in a spreadsheet file, see the Excel Case Converter for fixing the case of a whole column first.
A spreadsheet has its own command for this: in Excel, select the column and use Data, then Remove Duplicates. That works on whole rows or chosen columns. This tool is for plain text, where each line is one value. Once the repeats are gone, Sort Lines puts the list in order, Add Text to Each Line wraps every item in quotes or tags, and Column to Comma Separated List turns the list into one line. To see which items two lists share and which only one has, use Compare Two Lists.
Remove Duplicate Lines in Code and on the Command Line
If you do this often inside a script, these one-liners do the same job. The first keeps the original order, like this tool. The second sorts the lines and so loses the order.
awk '!seen[$0]++' list.txt > unique.txt
sort -u list.txt > unique.txtlines = open("list.txt", encoding="utf-8").read().splitlines()
unique = list(dict.fromkeys(lines))
print("\n".join(unique))In Python, a dictionary remembers insertion order, so dict.fromkeys drops repeats and keeps the first copy of each line.
Limits and Things to Know
- Lines are compared exactly, character for character. A letter with an accent and the same letter without it are different lines, and so are two visually identical letters that are built from different Unicode characters.
- Case matching is done with the browser's standard lowercase rule. It is not a language-aware comparison.
- Windows line endings are converted to plain line breaks when text is loaded, so the result uses one kind of line break throughout.
- A trailing line break at the end of the text counts as a blank line. Tick Remove empty lines if you do not want it.
- The tool does not detect near duplicates such as lines with a typo. Only identical lines, after your chosen options, are removed.
- Input is limited to 2,000,000 characters. Very long lists may take a moment to show.
Your List Stays in Your Browser
The lines are compared by code running in your browser. They are not sent to a server and are not stored. See the Privacy Policy for details. Closing or reloading the page clears the text.