Keeping Linux performance history with atop and PCP
Record and review Linux server performance history with atop, then use PCP archives (pmlogger, pmrep, pmiostat) on Red Hat.
Two tools keep a history of system performance on Linux: atop, which shows system activity and can write raw logs to review later, and PCP (Performance Co-Pilot), which is pre-configured on Red Hat systems to archive metrics.
atop
Install and enable (RHEL, CentOS):
sudo yum install atop
sudo systemctl enable --now atop
The recording interval is set in the daily collection configuration:
| Distribution | File to edit |
|---|---|
| CentOS 6 | /etc/atop/atop.daily |
| CentOS 7 | /etc/sysconfig/atop or /usr/share/atop/atop.daily |
| Arch Linux | /etc/default/atop |
Record a sample every 10 seconds by hand:
atop -w /var/log/atop/atop_20210320 10
Read a recorded file:
atop -r /var/log/atop/atop_20210221
Rotate the logs with a systemd timer:
systemctl enable --now atop-rotate.timer
systemctl status atop-rotate.timer
The optional netatop kernel module adds per-process network statistics:
yum update netatop
sudo modprobe netatop
PCP (Red Hat)
Install:
sudo dnf -y install pcp-zeroconf
sudo dnf -y install pcp-system-tools pcp-doc
The pcp-zeroconf package pre-configures pmcd, pmlogger and pmie. By default it captures CPU, network, file system, disk and memory metrics.
Locations and configuration:
- archives:
/var/log/pcp/pmlogger/<host>/; - control file:
/etc/pcp/pmlogger/control.d/local; - retention:
/etc/sysconfig/pmlogger_timers; - choosing metrics:
pmlogconf /var/lib/pcp/config/pmlogger/config.default.
Inspect an archive:
ls -altr /var/log/pcp/pmlogger/<host>/
pmdumplog -L /var/log/pcp/pmlogger/<host>/20230914.00.10.2
Copy an archive to keep it:
pmlogextract 20230914.00.10.2 ~/pcp_archive/
Use the data:
pmrep -z -p -t 10 -a ~/pcp_archive kernel.all.load kernel.all.cpu.{user,sys,wait.total} disk.all.total_bytes
Disk I/O from an archive, starting at 20:55:00, 10 samples 10 minutes apart, for vg-* volumes:
pmiostat -z -a /var/log/pcp/pmlogger/<host>/20230916 -x t -x dm -t 10m -S'@20:55:00' -s 10 -P0 -R'vg-*'
Notes
- Paths, package names and options vary with the distribution and the PCP version.
<host>replaces the machine name in the archive paths.