Linux command-line cheat sheet: find, du, grep, lsof, signals and archives
Linux command-line recipes: zombie processes and signals, lsof, copying, deleting and measuring files with find and du, grep, tar, zip.
A cheat sheet of Linux command-line recipes for everyday administration: processes and signals, open files, file management with find, du, grep, archives and a few text clean-up tricks.
Processes and network
ps -auxwf # process tree, full command lines
netstat -tulpn # listening sockets with owning process
netstat -natu | grep 'ESTABLISHED'
sudo ss -lutn | grep 1521 # who listens on a given port
lsof -i -nP # network connections per process
Other useful tools: mtr (traceroute and ping combined), drill (DNS lookups; replaces the deprecated dig on some distributions), ss, free, du --max-depth=1.
Remove the last command from the shell history:
ls ; history -d $(history 1)
Build arguments with xargs:
echo "a b c" | xargs printf -- '-f %s\n'
Process states and zombies
The process state shown by ps is one of:
| State | Meaning |
|---|---|
| R | Running or runnable (waiting for a CPU slot) |
| S | Sleeping, waiting for an event such as I/O or a resource |
| D | Uninterruptible sleep: blocked in a system call, does not react to signals until it returns |
| T | Stopped by SIGSTOP; only reacts to SIGKILL or SIGCONT (this is what fg/bg job control uses) |
| Z | Zombie: the process finished but its process table entry remains until the parent reads its exit status (the parent receives SIGCHLD) |
A zombie cannot be killed itself; identify its parent and act on the parent:
ps aux | egrep "Z|defunct"
ps -o ppid= -p <PID>
kill -SIGKILL <PARENT_PID>
To stop all processes of a user, lsof -t -u <user> lists their PIDs, which can be passed to kill -9.
Common signals
| Signal | Number | Description |
|---|---|---|
| SIGHUP | 1 | Controlling terminal closed or controlling process ended |
| SIGINT | 2 | Sent by Ctrl-C |
| SIGQUIT | 3 | Sent by Ctrl-\ |
| SIGFPE | 8 | Illegal arithmetic operation |
| SIGKILL | 9 | Immediate termination, cannot be caught |
| SIGALRM | 14 | Alarm clock |
| SIGTERM | 15 | Polite request to terminate |
| SIGCONT | 18 | Resume after a stop; sent by bg and fg |
| SIGSTOP | 19 | Stop the process, cannot be caught |
| SIGTSTP | 20 | Terminal stop (Ctrl-Z); the program may ignore it |
Signal numbers vary by architecture; see the list of signals.
Open files with lsof
lsof /var/log/syslog # who has this file open
lsof +D /var/log/ # who has files open in this directory
Recover a deleted file that is still held open by a process (the process stays alive, the file is reachable through /proc):
less test.txt # then Ctrl-Z to send it to the background
lsof | grep test.txt # get the PID
cd /proc/<PID>/fd/
ls
cp <fd> /safe/place/test.txt # copy the file descriptor
System information
cat /etc/*release* # distribution and release
32 or 64 bit:
uname -a
dpkg --print-architecture
getconf LONG_BIT
arch
file /sbin/init
i386 means 32 bit, amd64 means 64 bit.
Listing hidden files
ls -d .*
ls -d .b*
ls -d .[!.]*
Build the wildcard so that . and .. do not match:
grep -l 'PATH' ~/.[!.]*
Copy and move
Copy everything in the current directory to /target, preserving ownership, permissions and timestamps:
tar cf - * | ( cd /target; tar xfp -)
Plain copy of files and directories:
cp -rp source destination
| Option | Long name | Description |
|---|---|---|
| -a | --archive | Same as -dR --preserve=all |
| -p | Same as --preserve=mode,ownership,timestamps |
|
| -r | --recursive | Copy directories recursively |
| -l | --link | Hard-link files instead of copying |
| -s | --symbolic-link | Create symbolic links instead of copying |
Copy over the network with tar through ssh:
tar -cf - /some/file | ssh host.name tar -xf - -C /destination
# with compression
tar -czf - /some/file | ssh host.name tar -xzf - -C /destination
Copy a directory structure and link the files instead of copying them:
cp -as SOURCE/ COPY
cp -as "$(pwd)/SOURCE/" COPY
cp -al /src/dir /dest/dir # hard links to leaf files
cp -as /src/dir /dest/dir # symlinks to leaf files
cp -rl /src/dir /dest/dir # hard links to leaf files
cp -rs /src/dir /dest/dir # symlinks to leaf files
Delete
Always run the command with -print (or without -exec) first to check what will be deleted.
Directories older than 7 days (empty ones only, rmdir refuses otherwise):
find . -mtime +7 -type d -exec rmdir {} \;
Empty directories, recursively:
# check
find . -type d -empty -print
# delete
find . -type d -empty -delete
or
find . -type d -empty
sudo find . -type d -empty -exec rmdir {} \;
Old files:
find /opt/backup -type f -mtime +30
find /opt/backup -type f -mtime +30 -exec rm -f {} \;
find /var/log -name "*.log" -type f -mtime +30
find /var/log -name "*.log" -type f -mtime +30 -exec rm -f {} \;
Delete everything except the most recent files (! -newer selects files that are not newer than the reference file; the reference is the newest file to delete):
find magick-* ! -newer <reference-file> -exec rm -f {} \; -print
find *.dbf ! -newer reference.dbf -exec stat {} --format "%n %y" \; # check
find *.dbf ! -newer reference.dbf -exec rm -f {} \; -print
Delete directories by name:
find . -type d -name "bar" -exec rm -rf "{}" \;
Delete all files with a given extension:
find . -name '*.zip' -print # check first
find . -name "*.zip" -type f -delete
find . -name "*.bak" -type f -print0 | xargs -0 /bin/rm -f
find . -name "*.bak" -type f -exec rm -f {} \; -print
Measure files and sizes
Total size of files matching a pattern and age:
find . -name "FILEPREFIX*" -type f -mtime +3 -exec du -ch {} + | grep total$
Number of files per day:
find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c
Size in GB per day:
find . -type f -printf '%TY-%Tm-%Td %s\n' | awk '
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%0.10f \n", i, a[i]/1024/1024/1024 | " sort -nk1"; }
}'
Total size of all files with a given extension in a directory tree (source discussion):
find . -name "*.log.*" -print0 | du -sb --files0-from=- | awk '{ total += $1} END { print total/1024/1024/1024 }'
find . -name "*.o" | xargs du -sch
find . -name "*.jpg" -print0 | xargs -0 du -sch # -print0 handles spaces in file names
find . -name "*.jpg" -printf '%s\n' | awk '{c+=$1}END{print c}'
find . -name "*.jpg" -exec du -b {} \; | awk '{total+=$1}END{print total}'
shopt -s globstar
du -sch **/*.jpg
Total size per extension:
#!/bin/bash
ftypes=$(find . -type f | grep -E ".*\.[a-zA-Z0-9]*$" | sed -e 's/.*\(\.[a-zA-Z0-9]*\)$/\1/' | sort | uniq)
for ft in $ftypes
do
echo -n "$ft "
find . -name "*${ft}" -exec ls -l {} \; | awk '{total += $5} END {print total}'
done
Size for a single extension:
find /path -name '*.frq' -exec ls -l {} \; | awk '{ Total += $5 } END { print Total }'
find /var/app -type f -name '*.aud' -exec du -ch {} + | grep total$
Largest files and folders
sudo du -aBm / 2>/dev/null | sort -nr | head -n 10
sudo du --max-depth=1 # size by folder (Linux)
du | grep -v '/.*/' # size by folder (AIX)
du | awk -F/ 'NF <= 2' # size by folder (AIX)
du -hms /home/user/* | sort -nr | head
sudo du -a /dir/ | sort -n -r | head -n 20
sudo find / -type f -printf "%s\t%p\t\n" | sort -n | tail -1
find $HOME -type f -printf '%s %p\n' | sort -nr | head -10
Count files per folder
# files and directories (inode count) per top-level folder
du -a 2>/dev/null | cut -d/ -f2 | sort | uniq -c | sort -nr
# only files
find . -type f | cut -d/ -f2 | sort | uniq -c
# files per directory
find . -type d -print0 | while read -d '' -r dir; do
files=("$dir"/*)
printf "%5d files in directory %s\n" "${#files[@]}" "$dir"
done
# totals
find / -type d -printf "d\n" -o -type f -printf "f\n" | awk '/f/{f++}/d/{d++}END{printf "%d directories, %d files\n", d, f}'
find / -printf "%y\n" | awk '/f/{f++}/d/{d++}END{printf "%d directories, %d files\n", d, f}'
Archives
tar
# create
tar -cvzf archive.tar.gz file_or_directory ... # gzip
tar -cvzf - /var/log/* > varlog.tar.gz
tar -cvjf archive.tar.bz2 file_or_directory ... # bzip2
tar -C from/ -cvf test.tar path/sub/
# list and extract
tar -tvf archive.tar
tar -xvzf archive.tar.gz
tar -xvjf archive.tar.bz2
tar -C to/ -xvf test.tar
zip
Unzip every archive found in a directory tree, each into a folder named after the archive (see also find -exec vs find | xargs):
find . -name '*.zip' | while read filename; do unzip -o -d "$(dirname "${filename}")/$(basename -s .zip "${filename}")" "$filename" ; done;
In parallel (replace <N> with the number of parallel processes):
find . -name "*.zip" | xargs -P <N> -I fileName sh -c 'unzip -o -d "$(dirname "fileName")/$(basename -s .zip "fileName")" "fileName"'
Zip each sub-folder into its own archive, sequentially or in parallel (-0 stores without compression):
for i in */; do zip -r "${i%/}.zip" "$i"; done
for i in */; do zip -0 -r "${i%/}.zip" "$i" & done; wait
Search
Files by name, biggest last:
find . -name "*.data*" -exec du -m {} \; | sort -n
Size per extension (reference):
for ext in `find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u`; do
echo $ext": "`find . -name "*."$ext -print0 | du -ch --files0-from=- | tail -1`
done
Files newer or older than a date:
find /dir/ ! -type f -newerXY 'yyyy-mm-dd'
find . -name 'app.log*' -mtime +2 -exec stat {} --format "%n %y" \;
# files modified since yesterday
find /directory -newermt $(date +%Y-%m-%d -d '1 day ago') -type f -print
Search file contents for a string in smaller files (-size -100000 is in 512-byte blocks):
find / -type f -size -100000 -exec grep -il 'PATTERN' {} \; 2> /dev/null
Search inside files with grep
grep --include=\*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"
grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"
grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"
grep -rnw '/path/to/somewhere/' -e 'pattern'
-rrecursive,-nline numbers,-wwhole word,-lonly the names of matching files.- A recursive search from
/can take a very long time.
Several patterns in a log:
egrep -w 'warning|error|critical' /var/log/messages
grep 'warning\|error\|critical' /var/log/messages
egrep -wi --color 'warning|error|critical' /var/log/messages
Filter out the lines listed in a text file:
grep ORA- alert.log | grep -v "`cat exclude.txt`"
Search only the end of a very large file (much faster than reading it all):
dd if=file bs=1 skip=14G | grep xxx # under 1 second
tail -c 14g file | grep xxx # under 2 seconds
Extract and count IP addresses from a log:
grep -E -o "^[^^][0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" access.log | uniq -c | more
Example output: 42 192.0.2.10, 16 192.0.2.11.
Compare two file lists and show the modification date of the files only in the first:
comm -1 list_a.lst list_b.lst | xargs -n1 stat --format "%n %y"
Miscellaneous
Free space from the last line of df, and file names from ls -la:
df --total | tail -n 1 | awk '{print $4}'
ls -la | awk '{print $9}'
Text file clean-up
# remove DOS carriage returns
sed -i 's/\r$//' filename
# remove the space before a colon (YAML)
find env/ -type f -exec sed -i -e 's/ :/:/g' {} \;
# remove trailing spaces
find env/ -type f -exec sed -i -e 's/[[:space:]]*$//' {} \;
# remove empty lines
find env/ -type f -exec sed -i -e '/^$/d' {} \;
Shutdown and boot targets
Equivalent commands (see this discussion):
| systemd target | Equivalent commands |
|---|---|
systemctl isolate halt.target |
shutdown -H now, systemctl halt, halt |
systemctl isolate reboot.target |
shutdown -r now, telinit 6, systemctl reboot, reboot |
systemctl isolate poweroff.target |
shutdown -P now, telinit 0, shutdown now, systemctl poweroff, poweroff |
systemctl isolate rescue.target |
telinit 1, systemctl rescue |
systemctl isolate multi-user.target |
telinit 2, telinit 3, telinit 4 |
systemctl isolate graphical.target |
telinit 5 |
Notes
- Always run a destructive
find ... -exec rmor-deleteas a dry run (-print) first. - Some commands scan the whole filesystem (
/) and can be slow and I/O heavy on production systems.