Linux service management cheat sheet: systemd, SysV init and OpenRC
systemctl, journalctl, systemd-analyze, chkconfig and rc-update commands to manage, analyse and troubleshoot Linux services.
Cheat sheet for managing Linux services with systemd, with a reminder of the SysV init (chkconfig, service) and OpenRC equivalents.
systemd
Unit file locations
| Path | Description |
|---|---|
/etc/systemd/system |
Local configuration |
/run/systemd/system |
Runtime units |
/usr/lib/systemd/system |
Units of installed packages |
/usr/lib/systemd/system should only contain unit files placed there by the package manager (YUM/DNF/RPM/APT...). Files in /etc/systemd/system are placed manually by the administrator (tarball installations, home-grown scripts).
Applying a change:
systemctl daemon-reload
systemctl reload unit
systemctl start name.service
Unit types
See Understanding Systemd Units and Unit Files.
| Type | Purpose |
|---|---|
.service |
Manages a service or application (start, stop, dependencies, ordering) |
.socket |
Network/IPC socket or FIFO for socket activation; starts the associated .service |
.device |
Device managed by systemd through udev or sysfs |
.mount |
Mount point (name = path with / replaced by -); can be generated from /etc/fstab |
.automount |
Automatically mounted mount point, with a matching .mount |
.swap |
Swap space |
.target |
Synchronization point at boot or when changing state |
.path |
Path-based activation (inotify) |
.timer |
Scheduled or delayed activation, similar to cron |
.snapshot |
Created by systemctl snapshot to restore a temporary state |
.slice |
cgroup node used to limit or assign resources |
.scope |
Externally created set of processes, managed automatically |
systemctl status *timer
systemctl status *.mount*
systemctl status *swap
Common commands
systemctl start [name.service]
systemctl stop [name.service]
systemctl restart [name.service]
systemctl reload [name.service]
systemctl status [name.service]
systemctl is-active [name.service]
systemctl list-units --type=service --all
systemctl list-units --type=target --all
systemctl list-units -t service
systemctl list-unit-files
systemctl --failed
Boot analysis
pstree
systemctl list-units --type=target
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain
sudo systemctl list-dependencies
systemd-analyze critical-chain --fuzz 1h
systemd-analyze critical-chain network.target local-fs.target
systemd-analyze plot > boot_time.svg
See also systemd critical-chain.
Inspecting and modifying a unit
systemctl show myapp.service
systemctl show myapp.service --property=ExecStart
systemctl cat myapp.service
systemctl edit myapp.service
systemctl edit --runtime myapp.service
Typical locations: the original file in /usr/lib/systemd/system/myapp.service and overrides in /etc/systemd/system/myapp.service.d/. A --runtime change is written to /run/systemd/system/<unit>.d/override.conf.
systemd-delta | egrep -v '.scope|.slice'
systemd-delta --type extended /etc/systemd/system/
systemctl daemon-reload
To create a service, see Create a systemd service in Linux.
Targets and runlevel equivalents
| Runlevel | Legacy name | Target |
|---|---|---|
| 0 | runlevel0.target | poweroff.target |
| 1 | s, single, runlevel1.target | rescue.target |
| 2 | runlevel2.target | multi-user.target |
| 3 | runlevel3.target | multi-user.target |
| 4 | runlevel4.target | multi-user.target |
| 5 | runlevel5.target | graphical.target |
| 6 | runlevel6.target | reboot.target |
| 7 | emergency | emergency.target |
systemctl list-units --type target --state active
systemctl get-default
systemctl isolate graphical.target
systemctl isolate multi-user.target
systemctl set-default graphical.target
Miscellaneous
hostnamectl
timedatectl status
systemctl restart systemd-timesyncd
Journal (journalctl)
journalctl -f is the equivalent of tail -f /var/log/messages. Classic log files: Red Hat /var/log/messages and /var/log/secure; Debian family /var/log/syslog and /var/log/auth.log.
Cleanup and persistence
journalctl --disk-usage
journalctl --list-boots
sudo journalctl --vacuum-size=50M
sudo journalctl --vacuum-time=4weeks
egrep -v '^#' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
Settings in /etc/systemd/journald.conf:
Storage=auto
SystemMaxUse=500M
MaxFileSec=5day
SystemMaxFileSize=400M
Filtering and searching
journalctl -b
journalctl --since "2015-01-10 17:15:00"
journalctl --since yesterday
journalctl --since 09:00 --until "1 hour ago"
journalctl -u nginx.service
journalctl _PID=8088
journalctl -n 20
journalctl _UID=108
journalctl -t sudo
journalctl -t sudo -o verbose _PID=1959 _CMDLINE="sudo ls"
journalctl -g denied*
journalctl --boot -o verbose
journalctl --boot _TRANSPORT=audit
journalctl _SYSTEMD_UNIT=sshd.service | egrep "Accepted"
journalctl AUDIT_FIELD_RES=failed -o verbose
journalctl -u apache2.service -r -o json-pretty
journalctl -o json | jq 'select(.MESSAGE | test("failed|failure"; "i")) | .MESSAGE'
Write to the journal with an identifier, then follow it:
systemd-cat -t myapp echo "test"
journalctl -f -t myapp
Priorities (-p option):
| Value | Level |
|---|---|
| 0 | emerg |
| 1 | alert |
| 2 | crit |
| 3 | err |
| 4 | warning |
| 5 | notice |
| 6 | info |
| 7 | debug |
journalctl -p err
journalctl -b -p err
Start and stop history of a service
journalctl SYSLOG_IDENTIFIER=systemd
journalctl | grep "Starting" | wc -l
Counting the restarts of a unit:
journalctl --since=<date> --until=<another date> UNIT=<service name> SYSLOG_IDENTIFIER=systemd |
awk '
/Started/{ started = 1 }
/Stopped/{ if (started) { started = 0; restarted++; } }
END {
print "It was restarted " restarted "times."
}
'
SysV init (chkconfig / service)
chkconfig service_name on
chkconfig service_name on --level 2,3,4,5
chkconfig service_name off
chkconfig --list
chkconfig --list service_name
chkconfig --list | grep 5:on
chkconfig --del service_name
chkconfig service_name reset
ls /etc/init.d/*
service service_name status
service --status-all
service service_name start
service service_name stop
sudo /etc/init.d/atop restart
CentOS 5/6 example (SSH)
See CentOS 6: install, start, stop, enable and disable SSH server.
# Start
sudo service sshd start
/etc/init.d/sshd start
# Stop
sudo service sshd stop
# Enable at boot
sudo chkconfig sshd --add
sudo chkconfig sshd on --level 2,3,4,5
# Disable
sudo chkconfig sshd off
sudo chkconfig sshd --del
OpenRC
OpenRC uses runlevels much like sysvinit. There are three internal runlevels (sysinit, shutdown, reboot) and user runlevels: boot (services required by the other runlevels), default (day-to-day operation), nonetwork (no network needed) and single (single-user mode). Transition: sysinit -> boot -> default -> shutdown.
The services of a runlevel are controlled via /etc/inittab and the rc-update command.
rc-update -v
rc-update add <service> [runlevel]
rc-update del <service> [runlevel]