LLI Wiki — My Digital Knowledge Base

Understanding and managing the Linux OOM killer

Find the processes the kernel killed for lack of memory, read and adjust their OOM score, and choose the allocation behaviour.

The Linux kernel kills a process when memory runs out (the OOM killer, for out of memory). This article explains how to find the killed processes, read their score and adjust the behaviour.

Find the killed processes

journalctl -a | grep -i "killed process"

Per boot:

journalctl --list-boots | \
  awk '{ print $1 }' | \
  xargs -I{} journalctl --utc --no-pager -b {} -kqg 'killed process' -o verbose --output-fields=MESSAGE

Score of a process

cat /proc/$PID/oom_score

The score is adjusted with oom_score_adj (from -1000 to 1000). A negative value makes the process less likely to be killed; -1000 protects it completely.

echo -100 > /proc/$PID/oom_score_adj

The older oom_adj interface (from -17 to 15) is deprecated but still present:

Action Command
Read the value cat /proc/$PID/oom_adj
Make the process more likely to be killed echo 15 > /proc/$PID/oom_adj
Make the process less likely to be killed echo -15 > /proc/$PID/oom_adj

Kernel settings

sudo sysctl -a | grep oom

Memory allocation is controlled by vm.overcommit_memory (in /etc/sysctl.conf or a file in /etc/sysctl.d/):

Value Behaviour
0 Heuristic: estimates whether there is enough memory (default)
1 Always accepts the allocation
2 Refuses the allocation when there is not enough memory

To choose which process is killed when memory runs out, vm.oom_kill_allocating_task defaults to 0 (the process with the highest score is killed); 1 kills the process that caused the shortage.

Protect a service with systemd

In the [Service] section of the unit (for example through systemctl edit <service>):

[Service]
OOMScoreAdjust=-1000

Watch the OOM candidates

dstat --top-oom

Shows the process most likely to be killed.

Remarks

Mastodon LLIMastodon SocialFosstodon