Linux: remove duplicate lines without sorting using awk
Deduplicate lines of one or more files with awk while preserving the original order.
To remove duplicate lines from a file or stream while keeping the original order (without sort | uniq), awk is enough.
awk '!x[$0]++'
awk '!a[$0]++' file1.csv file2.csv
How it works
$0is the current line;x[$0]is an associative array indexed by the line content.x[$0]++returns the current value (0 the first time) and then increments it;!negates the result.- The condition is therefore true only the first time a line is seen, and awk's default action (print the line) then runs.
- With several input files, deduplication applies across all of them.
Notes
- Line order is preserved, unlike
sort -u. - Every unique line is kept in memory: keep an eye on this with very large files.