LLI Wiki — My Digital Knowledge Base

Remove duplicate lines without sorting (Python)

A short Python filter that removes duplicate lines from standard input while keeping the original order.

This short Python filter reads lines from standard input, writes each line the first time it appears, and sets repeated lines aside. Nothing is sorted, so the original order is kept.

Script

#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
  if line in unique_lines:
    duplicate_lines.append(line)
  else:
    unique_lines.append(line)
    sys.stdout.write(line)
# optionally do something with duplicate_lines

Usage

Run the script as a filter that reads standard input, for example python script.py < input.txt > output.txt.

Notes

Mastodon LLIMastodon SocialFosstodon