CPU and Load¶

Load average and per-CPU statistics tell whether a host is compute-bound, and if so which processes are responsible. Reading us, sy, wa and st correctly separates a real CPU limit from I/O, kernel work or a noisy cloud neighbour.

Track: Core · Interview weight: High

Must-Know Facts¶

Load Average, Really¶

Load average is the number of tasks that are either running or waiting to run, plus those in uninterruptible sleep, averaged over one, five and fifteen minutes. Compared against the CPU count it shows headroom: a load of 2.0 on two CPUs is fully used, and the same load on eight CPUs is quiet.

nproc
cat /proc/loadavg

Output:

2
1.02 0.24 0.08 4/100 1509

The three averages here rise from 0.08 to 1.02, so demand is climbing. The 4/100 field is running over total tasks, and the last number is the most recent PID.

The vmstat CPU Columns¶

vmstat splits CPU time into us, sy, id, wa and st, and shows the run queue in r. A run queue above the core count with high us is the clearest sign of a compute limit.

vmstat 1 3

Output:

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 4  0      0 343284  10780 430336    0    0   681   710  127    0  2  1 97  0  0  0
 3  0      0 343284  10780 430336    0    0     0     0  511  168 100  0  0  0  0  0
 3  0      0 343284  10780 430336    0    0     0     0  509  162 100  0  0  0  0  0

With r at 3 to 4 on two CPUs and us at 100, this host is CPU-bound in user code. If sy were high instead, the load would be kernel work; if wa were high, the disk would be the limit despite the busy CPU line.

Per-CPU Balance with mpstat¶

A total that reads 50% busy can hide one saturated core and one idle core, which happens with single-threaded work. mpstat -P ALL breaks the total into per-CPU rows.

mpstat -P ALL 1 1

Output:

22:59:51     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
22:59:52     all  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
22:59:52       0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
22:59:52       1  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

Both CPUs sit at 100% user here, so the work parallelises. A single hot core with the rest idle would point at a single-threaded bottleneck that more cores cannot help.

Per-Process CPU with pidstat and top¶

Once the host is known to be CPU-bound, the next question is which process. pidstat attributes CPU per process each interval, including %wait, the time a runnable task spent off-CPU waiting for a core.

pidstat 1 1

Output:

23:00:11        0      1464   59.41    0.00    0.00   38.61   59.41     0  stress-ng-cpu
23:00:11        0      1465   77.23    0.00    0.00   22.77   77.23     1  stress-ng-cpu
23:00:11        0      1466   61.39    0.00    0.00   38.61   61.39     1  stress-ng-cpu

Three CPU-bound threads share two CPUs, so each runs 59 to 77% of the time and waits (%wait) the rest for a core. A top snapshot sorted by CPU shows the same processes with per-core percentages that can exceed 100 on multiple cores.

top -b -n1 -o %CPU | head -8

Output:

top - 23:00:11 up 5 min,  0 user,  load average: 1.02, 0.24, 0.08
Tasks:  95 total,   4 running,  91 sleeping,   0 stopped,   0 zombie
%Cpu(s): 95.2 us,  4.8 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1466 root      20   0   66176   7076   4228 R  90.0   0.7   0:18.20 stress-+
   1464 root      20   0   66176   7048   4228 R  60.0   0.7   0:18.34 stress-+

Context Switches and High System Time¶

High sy time or a high context-switch rate means the CPU is busy with kernel work, not application progress. pidstat -w separates voluntary switches (a task blocked and yielded) from involuntary ones (the scheduler preempted it).

pidstat -w 1 1

Output:

23:07:48      UID       PID   cswch/s nvcswch/s  Command
23:07:49        0        15      7.00      0.00  rcu_preempt
23:07:49        0        16      1.00      0.00  migration/0
23:07:49        0        20      1.00      0.00  migration/1

Many involuntary switches (nvcswch/s) point at more runnable threads than cores, while many voluntary switches (cswch/s) point at frequent blocking, often on locks or I/O. strace -c then shows which syscall dominates.

Steal and cgroup Throttling¶

On a shared host, st (steal) is time the hypervisor ran another guest instead of this one, so the CPU line looks idle while work stalls. Inside a container or systemd slice, a cpu.max quota throttles the group even when the host has spare CPU.

cat /sys/fs/cgroup/system.slice/cpu.max      # "quota period" or "max period"
grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/system.slice/cpu.stat

Persistent steal means a noisy neighbour or an oversubscribed instance; rising nr_throttled means the cgroup quota is too tight for the workload. Both look like a slow application while the host CPU appears free.

Common Errors¶

load average high but top shows CPUs mostly idle¶

Cause: tasks in D (uninterruptible) state count toward load; the bottleneck is I/O, not CPU.

Fix: check vmstat b and wa and iostat -xz 1; find the blocked tasks with ps -eo state,pid,cmd | grep '^D'.

%steal consistently above a few percent¶

Cause: the hypervisor is scheduling other guests onto this vCPU; the instance is oversubscribed or throttled.

Fix: move to a dedicated or larger instance; steal is not fixable from inside the guest.

Interview Checkpoints¶

vmstat 1 3          # r > nproc, high us
pidstat 1 1         # per-process %CPU

Related¶

• Methodology: where CPU fits in the 60-second checklist
• Process States: why D state raises load
• Server Slow: CPU as one branch of a slow host
• Priority and Nice: changing how the scheduler shares CPU

Captured on Rocky Linux 10.2 on an iximiuz Labs FlexBox microVM, kernel 6.1.167, 2026-09.