Understanding and managing the Linux OOM killer
Find the processes the kernel killed for lack of memory, read and adjust their OOM score, and choose the allocation behaviour.
The Linux kernel kills a process when memory runs out (the OOM killer, for out of memory). This article explains how to find the killed processes, read their score and adjust the behaviour.
Find the killed processes
journalctl -a | grep -i "killed process"
Per boot:
journalctl --list-boots | \
awk '{ print $1 }' | \
xargs -I{} journalctl --utc --no-pager -b {} -kqg 'killed process' -o verbose --output-fields=MESSAGE
Score of a process
cat /proc/$PID/oom_score
The score is adjusted with oom_score_adj (from -1000 to 1000). A negative value makes the process less likely to be killed; -1000 protects it completely.
echo -100 > /proc/$PID/oom_score_adj
The older oom_adj interface (from -17 to 15) is deprecated but still present:
| Action | Command |
|---|---|
| Read the value | cat /proc/$PID/oom_adj |
| Make the process more likely to be killed | echo 15 > /proc/$PID/oom_adj |
| Make the process less likely to be killed | echo -15 > /proc/$PID/oom_adj |
Kernel settings
sudo sysctl -a | grep oom
Memory allocation is controlled by vm.overcommit_memory (in /etc/sysctl.conf or a file in /etc/sysctl.d/):
| Value | Behaviour |
|---|---|
| 0 | Heuristic: estimates whether there is enough memory (default) |
| 1 | Always accepts the allocation |
| 2 | Refuses the allocation when there is not enough memory |
To choose which process is killed when memory runs out, vm.oom_kill_allocating_task defaults to 0 (the process with the highest score is killed); 1 kills the process that caused the shortage.
Protect a service with systemd
In the [Service] section of the unit (for example through systemctl edit <service>):
[Service]
OOMScoreAdjust=-1000
Watch the OOM candidates
dstat --top-oom
Shows the process most likely to be killed.
Remarks
- The
rcctlcommand in the source note is specific to OpenBSD and is not reproduced. - The source examples showed the real host name in the prompt; the prompt is replaced with a generic one.