Remove duplicate lines without sorting (Python)
A short Python filter that removes duplicate lines from standard input while keeping the original order.
This short Python filter reads lines from standard input, writes each line the first time it appears, and sets repeated lines aside. Nothing is sorted, so the original order is kept.
Script
#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
if line in unique_lines:
duplicate_lines.append(line)
else:
unique_lines.append(line)
sys.stdout.write(line)
# optionally do something with duplicate_lines
Usage
Run the script as a filter that reads standard input, for example python script.py < input.txt > output.txt.
Notes
line in unique_linesscans the whole list for every line, so the run time grows with the square of the number of lines. For large files, use a set (seen = set()): membership tests are then constant-time and the output order stays the same.- Each line keeps its trailing newline. The last line of a file without a newline (
a) therefore does not match the same text followed by a newline (a\n). duplicate_linesis filled but not used by the script; it is there for optional processing.