LLI Wiki — My Digital Knowledge Base

Linux command-line cheat sheet: find, du, grep, lsof, signals and archives

Linux command-line recipes: zombie processes and signals, lsof, copying, deleting and measuring files with find and du, grep, tar, zip.

A cheat sheet of Linux command-line recipes for everyday administration: processes and signals, open files, file management with find, du, grep, archives and a few text clean-up tricks.

Processes and network

ps -auxwf                      # process tree, full command lines
netstat -tulpn                 # listening sockets with owning process
netstat -natu | grep 'ESTABLISHED'
sudo ss -lutn | grep 1521      # who listens on a given port
lsof -i -nP                    # network connections per process

Other useful tools: mtr (traceroute and ping combined), drill (DNS lookups; replaces the deprecated dig on some distributions), ss, free, du --max-depth=1.

Remove the last command from the shell history:

ls ; history -d $(history 1)

Build arguments with xargs:

echo "a b c" | xargs printf -- '-f %s\n'

Process states and zombies

The process state shown by ps is one of:

State Meaning
R Running or runnable (waiting for a CPU slot)
S Sleeping, waiting for an event such as I/O or a resource
D Uninterruptible sleep: blocked in a system call, does not react to signals until it returns
T Stopped by SIGSTOP; only reacts to SIGKILL or SIGCONT (this is what fg/bg job control uses)
Z Zombie: the process finished but its process table entry remains until the parent reads its exit status (the parent receives SIGCHLD)

A zombie cannot be killed itself; identify its parent and act on the parent:

ps aux | egrep "Z|defunct"
ps -o ppid= -p <PID>
kill -SIGKILL <PARENT_PID>

To stop all processes of a user, lsof -t -u <user> lists their PIDs, which can be passed to kill -9.

Common signals

Signal Number Description
SIGHUP 1 Controlling terminal closed or controlling process ended
SIGINT 2 Sent by Ctrl-C
SIGQUIT 3 Sent by Ctrl-\
SIGFPE 8 Illegal arithmetic operation
SIGKILL 9 Immediate termination, cannot be caught
SIGALRM 14 Alarm clock
SIGTERM 15 Polite request to terminate
SIGCONT 18 Resume after a stop; sent by bg and fg
SIGSTOP 19 Stop the process, cannot be caught
SIGTSTP 20 Terminal stop (Ctrl-Z); the program may ignore it

Signal numbers vary by architecture; see the list of signals.

Open files with lsof

lsof /var/log/syslog           # who has this file open
lsof +D /var/log/              # who has files open in this directory

Recover a deleted file that is still held open by a process (the process stays alive, the file is reachable through /proc):

less test.txt                  # then Ctrl-Z to send it to the background
lsof | grep test.txt           # get the PID
cd /proc/<PID>/fd/
ls
cp <fd> /safe/place/test.txt   # copy the file descriptor

System information

cat /etc/*release*             # distribution and release

32 or 64 bit:

uname -a
dpkg --print-architecture
getconf LONG_BIT
arch
file /sbin/init

i386 means 32 bit, amd64 means 64 bit.

Listing hidden files

ls -d .*
ls -d .b*
ls -d .[!.]*

Build the wildcard so that . and .. do not match:

grep -l 'PATH' ~/.[!.]*

Copy and move

Copy everything in the current directory to /target, preserving ownership, permissions and timestamps:

tar cf - * | ( cd /target; tar xfp -)

Plain copy of files and directories:

cp -rp source destination
Option Long name Description
-a --archive Same as -dR --preserve=all
-p Same as --preserve=mode,ownership,timestamps
-r --recursive Copy directories recursively
-l --link Hard-link files instead of copying
-s --symbolic-link Create symbolic links instead of copying

Copy over the network with tar through ssh:

tar -cf - /some/file | ssh host.name tar -xf - -C /destination
# with compression
tar -czf - /some/file | ssh host.name tar -xzf - -C /destination

Copy a directory structure and link the files instead of copying them:

cp -as SOURCE/ COPY
cp -as "$(pwd)/SOURCE/" COPY

cp -al /src/dir /dest/dir    # hard links to leaf files
cp -as /src/dir /dest/dir    # symlinks to leaf files

cp -rl /src/dir /dest/dir    # hard links to leaf files
cp -rs /src/dir /dest/dir    # symlinks to leaf files

Delete

Always run the command with -print (or without -exec) first to check what will be deleted.

Directories older than 7 days (empty ones only, rmdir refuses otherwise):

find . -mtime +7 -type d -exec rmdir {} \;

Empty directories, recursively:

# check
find . -type d -empty -print
# delete
find . -type d -empty -delete

or

find . -type d -empty
sudo find . -type d -empty -exec rmdir {} \;

Old files:

find /opt/backup -type f -mtime +30
find /opt/backup -type f -mtime +30 -exec rm -f {} \;
find /var/log -name "*.log" -type f -mtime +30
find /var/log -name "*.log" -type f -mtime +30 -exec rm -f {} \;

Delete everything except the most recent files (! -newer selects files that are not newer than the reference file; the reference is the newest file to delete):

find magick-* ! -newer <reference-file> -exec rm -f {} \; -print
find *.dbf ! -newer reference.dbf -exec stat {} --format "%n %y" \;   # check
find *.dbf ! -newer reference.dbf -exec rm -f {} \; -print

Delete directories by name:

find . -type d -name "bar" -exec rm -rf "{}" \;

Delete all files with a given extension:

find . -name '*.zip' -print      # check first
find . -name "*.zip" -type f -delete
find . -name "*.bak" -type f -print0 | xargs -0 /bin/rm -f
find . -name "*.bak" -type f -exec rm -f {} \; -print

Measure files and sizes

Total size of files matching a pattern and age:

find . -name "FILEPREFIX*" -type f -mtime +3 -exec du -ch {} + | grep total$

Number of files per day:

find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c

Size in GB per day:

find . -type f -printf '%TY-%Tm-%Td %s\n' | awk '
 { a[$1] += $2 }
 END {
   for (i in a) {
     printf "%-15s\t%0.10f \n", i, a[i]/1024/1024/1024 | " sort -nk1"; }
}'

Total size of all files with a given extension in a directory tree (source discussion):

find . -name "*.log.*" -print0 | du -sb --files0-from=- | awk '{ total += $1} END { print total/1024/1024/1024 }'
find . -name "*.o" | xargs du -sch
find . -name "*.jpg" -print0 | xargs -0 du -sch      # -print0 handles spaces in file names
find . -name "*.jpg" -printf '%s\n' | awk '{c+=$1}END{print c}'
find . -name "*.jpg" -exec du -b {} \; | awk '{total+=$1}END{print total}'

shopt -s globstar
du -sch **/*.jpg

Total size per extension:

#!/bin/bash
ftypes=$(find . -type f | grep -E ".*\.[a-zA-Z0-9]*$" | sed -e 's/.*\(\.[a-zA-Z0-9]*\)$/\1/' | sort | uniq)
for ft in $ftypes
do
    echo -n "$ft "
    find . -name "*${ft}" -exec ls -l {} \; | awk '{total += $5} END {print total}'
done

Size for a single extension:

find /path -name '*.frq' -exec ls -l {} \; | awk '{ Total += $5 } END { print Total }'
find /var/app -type f -name '*.aud' -exec du -ch {} + | grep total$

Largest files and folders

sudo du -aBm / 2>/dev/null | sort -nr | head -n 10
sudo du --max-depth=1                     # size by folder (Linux)
du | grep -v '/.*/'                       # size by folder (AIX)
du | awk -F/ 'NF <= 2'                    # size by folder (AIX)
du -hms /home/user/* | sort -nr | head
sudo du -a /dir/ | sort -n -r | head -n 20
sudo find / -type f -printf "%s\t%p\t\n" | sort -n | tail -1
find $HOME -type f -printf '%s %p\n' | sort -nr | head -10

Count files per folder

# files and directories (inode count) per top-level folder
du -a 2>/dev/null | cut -d/ -f2 | sort | uniq -c | sort -nr
# only files
find . -type f | cut -d/ -f2 | sort | uniq -c

# files per directory
find . -type d -print0 | while read -d '' -r dir; do
    files=("$dir"/*)
    printf "%5d files in directory %s\n" "${#files[@]}" "$dir"
done

# totals
find / -type d -printf "d\n" -o -type f -printf "f\n" | awk '/f/{f++}/d/{d++}END{printf "%d directories, %d files\n", d, f}'
find / -printf "%y\n" | awk '/f/{f++}/d/{d++}END{printf "%d directories, %d files\n", d, f}'

Archives

tar

# create
tar -cvzf archive.tar.gz file_or_directory ...        # gzip
tar -cvzf - /var/log/* > varlog.tar.gz
tar -cvjf archive.tar.bz2 file_or_directory ...       # bzip2
tar -C from/ -cvf test.tar path/sub/

# list and extract
tar -tvf archive.tar
tar -xvzf archive.tar.gz
tar -xvjf archive.tar.bz2
tar -C to/ -xvf test.tar

zip

Unzip every archive found in a directory tree, each into a folder named after the archive (see also find -exec vs find | xargs):

find . -name '*.zip' | while read filename; do unzip -o -d "$(dirname "${filename}")/$(basename -s .zip "${filename}")" "$filename" ; done;

In parallel (replace <N> with the number of parallel processes):

find . -name "*.zip" | xargs -P <N> -I fileName sh -c 'unzip -o -d "$(dirname "fileName")/$(basename -s .zip "fileName")" "fileName"'

Zip each sub-folder into its own archive, sequentially or in parallel (-0 stores without compression):

for i in */; do zip -r "${i%/}.zip" "$i"; done
for i in */; do zip -0 -r "${i%/}.zip" "$i" & done; wait

Files by name, biggest last:

find . -name "*.data*" -exec du -m {} \; | sort -n

Size per extension (reference):

for ext in `find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u`; do
    echo $ext": "`find . -name "*."$ext -print0 | du -ch --files0-from=- | tail -1`
done

Files newer or older than a date:

find /dir/ ! -type f -newerXY 'yyyy-mm-dd'
find . -name 'app.log*' -mtime +2 -exec stat {} --format "%n %y" \;
# files modified since yesterday
find /directory -newermt $(date +%Y-%m-%d -d '1 day ago') -type f -print

Search file contents for a string in smaller files (-size -100000 is in 512-byte blocks):

find / -type f -size -100000 -exec grep -il 'PATTERN' {} \; 2> /dev/null

Search inside files with grep

grep --include=\*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"
grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"
grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"
grep -rnw '/path/to/somewhere/' -e 'pattern'

Several patterns in a log:

egrep -w 'warning|error|critical' /var/log/messages
grep 'warning\|error\|critical' /var/log/messages
egrep -wi --color 'warning|error|critical' /var/log/messages

Filter out the lines listed in a text file:

grep ORA- alert.log | grep -v "`cat exclude.txt`"

Search only the end of a very large file (much faster than reading it all):

dd if=file bs=1 skip=14G | grep xxx     # under 1 second
tail -c 14g file | grep xxx             # under 2 seconds

Extract and count IP addresses from a log:

grep -E -o "^[^^][0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" access.log | uniq -c | more

Example output: 42 192.0.2.10, 16 192.0.2.11.

Compare two file lists and show the modification date of the files only in the first:

comm -1 list_a.lst list_b.lst | xargs -n1 stat --format "%n %y"

Miscellaneous

Free space from the last line of df, and file names from ls -la:

df --total | tail -n 1 | awk '{print $4}'
ls -la | awk '{print $9}'

Text file clean-up

# remove DOS carriage returns
sed -i 's/\r$//' filename
# remove the space before a colon (YAML)
find env/ -type f -exec sed -i -e 's/ :/:/g' {} \;
# remove trailing spaces
find env/ -type f -exec sed -i -e 's/[[:space:]]*$//' {} \;
# remove empty lines
find env/ -type f -exec sed -i -e '/^$/d' {} \;

Shutdown and boot targets

Equivalent commands (see this discussion):

systemd target Equivalent commands
systemctl isolate halt.target shutdown -H now, systemctl halt, halt
systemctl isolate reboot.target shutdown -r now, telinit 6, systemctl reboot, reboot
systemctl isolate poweroff.target shutdown -P now, telinit 0, shutdown now, systemctl poweroff, poweroff
systemctl isolate rescue.target telinit 1, systemctl rescue
systemctl isolate multi-user.target telinit 2, telinit 3, telinit 4
systemctl isolate graphical.target telinit 5

Notes

Mastodon LLIMastodon SocialFosstodon