LLI Wiki — My Digital Knowledge Base

Linux service management cheat sheet: systemd, SysV init and OpenRC

systemctl, journalctl, systemd-analyze, chkconfig and rc-update commands to manage, analyse and troubleshoot Linux services.

Cheat sheet for managing Linux services with systemd, with a reminder of the SysV init (chkconfig, service) and OpenRC equivalents.

systemd

Unit file locations

Path Description
/etc/systemd/system Local configuration
/run/systemd/system Runtime units
/usr/lib/systemd/system Units of installed packages

/usr/lib/systemd/system should only contain unit files placed there by the package manager (YUM/DNF/RPM/APT...). Files in /etc/systemd/system are placed manually by the administrator (tarball installations, home-grown scripts).

Applying a change:

systemctl daemon-reload
systemctl reload unit
systemctl start name.service

Unit types

See Understanding Systemd Units and Unit Files.

Type Purpose
.service Manages a service or application (start, stop, dependencies, ordering)
.socket Network/IPC socket or FIFO for socket activation; starts the associated .service
.device Device managed by systemd through udev or sysfs
.mount Mount point (name = path with / replaced by -); can be generated from /etc/fstab
.automount Automatically mounted mount point, with a matching .mount
.swap Swap space
.target Synchronization point at boot or when changing state
.path Path-based activation (inotify)
.timer Scheduled or delayed activation, similar to cron
.snapshot Created by systemctl snapshot to restore a temporary state
.slice cgroup node used to limit or assign resources
.scope Externally created set of processes, managed automatically
systemctl status *timer
systemctl status *.mount*
systemctl status *swap

Common commands

systemctl start [name.service]
systemctl stop [name.service]
systemctl restart [name.service]
systemctl reload [name.service]
systemctl status [name.service]
systemctl is-active [name.service]
systemctl list-units --type=service --all
systemctl list-units --type=target --all
systemctl list-units -t service
systemctl list-unit-files
systemctl --failed

Boot analysis

pstree
systemctl list-units --type=target
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain
sudo systemctl list-dependencies
systemd-analyze critical-chain --fuzz 1h
systemd-analyze critical-chain network.target local-fs.target
systemd-analyze plot > boot_time.svg

See also systemd critical-chain.

Inspecting and modifying a unit

systemctl show myapp.service
systemctl show myapp.service --property=ExecStart
systemctl cat myapp.service
systemctl edit myapp.service
systemctl edit --runtime myapp.service

Typical locations: the original file in /usr/lib/systemd/system/myapp.service and overrides in /etc/systemd/system/myapp.service.d/. A --runtime change is written to /run/systemd/system/<unit>.d/override.conf.

systemd-delta | egrep -v '.scope|.slice'
systemd-delta --type extended /etc/systemd/system/
systemctl daemon-reload

To create a service, see Create a systemd service in Linux.

Targets and runlevel equivalents

Runlevel Legacy name Target
0 runlevel0.target poweroff.target
1 s, single, runlevel1.target rescue.target
2 runlevel2.target multi-user.target
3 runlevel3.target multi-user.target
4 runlevel4.target multi-user.target
5 runlevel5.target graphical.target
6 runlevel6.target reboot.target
7 emergency emergency.target
systemctl list-units --type target --state active
systemctl get-default
systemctl isolate graphical.target
systemctl isolate multi-user.target
systemctl set-default graphical.target

Miscellaneous

hostnamectl
timedatectl status
systemctl restart systemd-timesyncd

Journal (journalctl)

journalctl -f is the equivalent of tail -f /var/log/messages. Classic log files: Red Hat /var/log/messages and /var/log/secure; Debian family /var/log/syslog and /var/log/auth.log.

Cleanup and persistence

journalctl --disk-usage
journalctl --list-boots
sudo journalctl --vacuum-size=50M
sudo journalctl --vacuum-time=4weeks
egrep -v '^#' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

Settings in /etc/systemd/journald.conf:

Storage=auto
SystemMaxUse=500M
MaxFileSec=5day
SystemMaxFileSize=400M

Filtering and searching

journalctl -b
journalctl --since "2015-01-10 17:15:00"
journalctl --since yesterday
journalctl --since 09:00 --until "1 hour ago"
journalctl -u nginx.service
journalctl _PID=8088
journalctl -n 20
journalctl _UID=108
journalctl -t sudo
journalctl -t sudo -o verbose _PID=1959 _CMDLINE="sudo ls"
journalctl -g denied*
journalctl --boot -o verbose
journalctl --boot _TRANSPORT=audit
journalctl _SYSTEMD_UNIT=sshd.service | egrep "Accepted"
journalctl AUDIT_FIELD_RES=failed -o verbose
journalctl -u apache2.service -r -o json-pretty
journalctl -o json | jq 'select(.MESSAGE | test("failed|failure"; "i")) | .MESSAGE'

Write to the journal with an identifier, then follow it:

systemd-cat -t myapp echo "test"
journalctl -f -t myapp

Priorities (-p option):

Value Level
0 emerg
1 alert
2 crit
3 err
4 warning
5 notice
6 info
7 debug
journalctl -p err
journalctl -b -p err

Start and stop history of a service

journalctl SYSLOG_IDENTIFIER=systemd
journalctl | grep "Starting" | wc -l

Counting the restarts of a unit:

journalctl --since=<date> --until=<another date> UNIT=<service name> SYSLOG_IDENTIFIER=systemd |
 awk '
      /Started/{ started = 1 }
      /Stopped/{ if (started) { started = 0; restarted++; } }
      END {
          print "It was restarted " restarted "times."
      }
 '

SysV init (chkconfig / service)

chkconfig service_name on
chkconfig service_name on --level 2,3,4,5
chkconfig service_name off
chkconfig --list
chkconfig --list service_name
chkconfig --list | grep 5:on
chkconfig --del service_name
chkconfig service_name reset

ls /etc/init.d/*
service service_name status
service --status-all
service service_name start
service service_name stop
sudo /etc/init.d/atop restart

CentOS 5/6 example (SSH)

See CentOS 6: install, start, stop, enable and disable SSH server.

# Start
sudo service sshd start
/etc/init.d/sshd start
# Stop
sudo service sshd stop
# Enable at boot
sudo chkconfig sshd --add
sudo chkconfig sshd on --level 2,3,4,5
# Disable
sudo chkconfig sshd off
sudo chkconfig sshd --del

OpenRC

OpenRC uses runlevels much like sysvinit. There are three internal runlevels (sysinit, shutdown, reboot) and user runlevels: boot (services required by the other runlevels), default (day-to-day operation), nonetwork (no network needed) and single (single-user mode). Transition: sysinit -> boot -> default -> shutdown.

The services of a runlevel are controlled via /etc/inittab and the rc-update command.

rc-update -v
rc-update add <service> [runlevel]
rc-update del <service> [runlevel]
Mastodon LLIMastodon SocialFosstodon