LLI Wiki — My Digital Knowledge Base

Keeping Linux performance history with atop and PCP

Record and review Linux server performance history with atop, then use PCP archives (pmlogger, pmrep, pmiostat) on Red Hat.

Two tools keep a history of system performance on Linux: atop, which shows system activity and can write raw logs to review later, and PCP (Performance Co-Pilot), which is pre-configured on Red Hat systems to archive metrics.

atop

Install and enable (RHEL, CentOS):

sudo yum install atop
sudo systemctl enable --now atop

The recording interval is set in the daily collection configuration:

Distribution File to edit
CentOS 6 /etc/atop/atop.daily
CentOS 7 /etc/sysconfig/atop or /usr/share/atop/atop.daily
Arch Linux /etc/default/atop

Record a sample every 10 seconds by hand:

atop -w /var/log/atop/atop_20210320 10

Read a recorded file:

atop -r /var/log/atop/atop_20210221

Rotate the logs with a systemd timer:

systemctl enable --now atop-rotate.timer
systemctl status atop-rotate.timer

The optional netatop kernel module adds per-process network statistics:

yum update netatop
sudo modprobe netatop

PCP (Red Hat)

Install:

sudo dnf -y install pcp-zeroconf
sudo dnf -y install pcp-system-tools pcp-doc

The pcp-zeroconf package pre-configures pmcd, pmlogger and pmie. By default it captures CPU, network, file system, disk and memory metrics.

Locations and configuration:

Inspect an archive:

ls -altr /var/log/pcp/pmlogger/<host>/
pmdumplog -L /var/log/pcp/pmlogger/<host>/20230914.00.10.2

Copy an archive to keep it:

pmlogextract 20230914.00.10.2 ~/pcp_archive/

Use the data:

pmrep -z -p -t 10 -a ~/pcp_archive kernel.all.load kernel.all.cpu.{user,sys,wait.total} disk.all.total_bytes

Disk I/O from an archive, starting at 20:55:00, 10 samples 10 minutes apart, for vg-* volumes:

pmiostat -z -a /var/log/pcp/pmlogger/<host>/20230916 -x t -x dm -t 10m -S'@20:55:00' -s 10 -P0 -R'vg-*'

Notes

Further reading

Mastodon LLIMastodon SocialFosstodon