LLI Wiki — My Digital Knowledge Base

Linux: remove duplicate lines without sorting using awk

Deduplicate lines of one or more files with awk while preserving the original order.

To remove duplicate lines from a file or stream while keeping the original order (without sort | uniq), awk is enough.

awk '!x[$0]++'
awk '!a[$0]++' file1.csv file2.csv

How it works

Notes

Mastodon LLIMastodon SocialFosstodon