Fix Linux High CPU Usage Load Average

Introduction

Linux high CPU usage and load average issues occur when system CPU resources are exhausted by runaway processes, inefficient applications, kernel bugs, or resource contention, causing system slowdown, unresponsiveness, and service degradation. Load average represents the average number of processes in runnable or uninterruptible sleep state over 1, 5, and 15-minute intervals. High load average with low CPU usage indicates I/O bottleneck, while high CPU with high load indicates CPU contention. Common causes include runaway process consuming CPU cycles, application memory leak causing GC thrashing, kernel bug or driver issue causing interrupt storm, misconfigured process affinity binding all processes to single core, container or VM CPU limits causing throttling, application deadlock causing spin-loop, excessive context switching from too many threads, I/O wait from slow storage or network, and kernel scheduling bugs under high concurrency. The fix requires understanding Linux process scheduling, CPU accounting, profiling tools (perf, top, htop, pidstat), cgroup resource control, and kernel debugging techniques. This guide provides production-proven troubleshooting for high CPU usage across RHEL/CentOS, Ubuntu/Debian, SUSE, and containerized environments.

Symptoms

top shows 100% CPU usage on one or more cores
Load average exceeds number of CPU cores
System responds slowly to commands
SSH connections lag or timeout
Application response time degrades
vmstat shows high r (runnable) queue
mpstat shows single core at 100%
Context switches per second extremely high
Interrupt rate abnormally high
Container/VM CPU throttled despite host capacity

Common Causes

Runaway process (infinite loop, excessive logging)
Java application GC thrashing (memory pressure)
Kernel interrupt storm (driver/hardware issue)
Excessive context switching (too many threads)
CPU throttling from cgroup limits
I/O wait causing process backlog
Application spin-lock (deadlock scenario)
NUMA memory locality issues
Process affinity misconfiguration
Kernel bug (soft lockup, RCU stall)

Step-by-Step Fix

### 1. Diagnose CPU usage

Check overall CPU usage:

```bash # Quick CPU overview top -bn1 | head -10

# Output: # %Cpu(s): 95.2 us, 2.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st # - us: user space (applications) # - sy: kernel space (system calls) # - ni: nice (low priority) # - id: idle (should be high when healthy) # - wa: I/O wait (high = storage bottleneck) # - hi: hardware interrupts # - si: software interrupts # - st: steal (VM waiting on hypervisor)

# Check load average uptime # load average: 15.23, 12.45, 8.67 # Values > CPU count indicate CPU contention

# Number of CPU cores nproc lscpu | grep "^CPU(s):" ```

Identify top CPU consumers:

```bash # Top processes by CPU top -bn1 -o %CPU | head -20

# Or use htop (more user-friendly) htop -C # Sorted by CPU

# Show CPU percentage per process ps aux --sort=-%cpu | head -20

# Key columns: # %CPU: CPU percentage # %MEM: Memory percentage # TIME: Total CPU time consumed # COMMAND: Process command

# Show threads sorted by CPU ps -eLo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | sort -k9 -rn | head -20 ```

Check per-core usage:

```bash # Per-CPU statistics mpstat -P ALL 1 5

# Output shows each core's usage: # CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle # all 95.00 0.00 2.00 0.00 0.00 1.00 0.00 0.00 0.00 2.00 # 0 98.00 0.00 1.00 0.00 0.00 0.50 0.00 0.00 0.00 0.50 # Saturated # 1 92.00 0.00 3.00 0.00 0.00 1.50 0.00 0.00 0.00 3.50

# Check if single core is bottleneck pidstat -u -t 1 5

# Shows per-thread CPU usage ```

### 2. Analyze runaway processes

Identify runaway process:

```bash # Find processes consuming excessive CPU ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -10

# Check process runtime ps -eo pid,etime,cmd --sort=-etime | head -10

# Long-running + high CPU = likely runaway

# Check process tree pstree -p <PID>

# See parent and child processes ```

Investigate process behavior:

```bash # Get process details ps -fp <PID>

# Check open files lsof -p <PID> | head -50

# Check network connections ss -tlnp | grep <PID> netstat -tlnp | grep <PID>

# Check process I/O pidstat -d -p <PID> 1 5

# Check process memory cat /proc/<PID>/status | grep -E "Vm|CPU"

# Check process threads ls -la /proc/<PID>/task/ cat /proc/<PID>/task/*/stat | awk '{print $1, $3, $14}' # PID, state, priority ```

Check for infinite loops:

```bash # Get thread-level CPU usage top -H -p <PID>

# Look for threads at 100% CPU

# Get stack traces of threads # For Java processes: jstack <PID> > /tmp/java-stack.txt

# For native processes, use perf: perf record -p <PID> -g -- sleep 10 perf report --stdio | head -100

# Look for tight loops in application code ```

### 3. Profile CPU usage with perf

Install perf:

```bash # Install perf apt install linux-tools-$(uname -r) # Debian/Ubuntu yum install perf # RHEL/CentOS

# Verify installation perf --version ```

Record CPU profile:

```bash # Record CPU usage (system-wide) perf record -a -g -o /tmp/perf.data sleep 30

# Record specific process perf record -p <PID> -g -o /tmp/perf-pid.data sleep 30

# Record with frequency (higher = more accurate, more overhead) perf record -F 99 -a -g -o /tmp/perf.data sleep 30

# Generate report perf report --stdio | head -100

# Or use interactive TUI perf report ```

Analyze perf output:

```bash # Show top functions by CPU time perf report --stdio -n | head -50

# Output example: # + 45.23% 12.34% myapp [.] process_request # + 23.45% 5.67% myapp [.] handle_connection # + 15.67% 2.34% libc [.] memcpy

# Find hotspots perf top # Real-time profiling

# Show call graph perf report --call-graph -n

# Annotate source code (if available) perf annotate process_request ```

Check for kernel issues:

```bash # Check for kernel lockups dmesg | grep -i "lockup\|rcu_stall"

# Check for soft lockups dmesg | grep "BUG: soft lockup"

# If found, CPU is stuck in kernel mode # May indicate driver bug or hardware issue

# Check interrupt rates cat /proc/interrupts | awk '{sum=0; for(i=2;i<=NF;i++) sum+=$i; print $0, sum}' | sort -kNF -rn | head -10

# High interrupt rate on single IRQ = potential issue ```

### 4. Check for I/O wait

Diagnose I/O wait:

```bash # Check I/O wait percentage iostat -x 1 5

# Look at %iowait column # High %iowait (>20%) indicates storage bottleneck

# Check disk utilization iostat -x 1 | grep -E "Device|sda"

# %util near 100% = disk saturated

# Check waiting processes vmstat 1 5

# Look at 'r' (runnable) and 'b' (blocked) columns # High 'b' = processes waiting on I/O

# Check I/O statistics per process pidstat -d 1 5 ```

Identify I/O-heavy processes:

```bash # Top I/O processes iotop -o -n 10

# Or with pidstat pidstat -d 1 | grep -A5 "Average:"

# Check process I/O wait for pid in /proc/[0-9]*/; do pid=$(basename $pid) io_wait=$(grep "blkio" /proc/$pid/io 2>/dev/null | awk '{print $2}') if [ -n "$io_wait" ] && [ "$io_wait" -gt 0 ]; then echo "PID $pid: $io_wait bytes read" fi done | sort -t: -k2 -rn | head -10 ```

Check storage performance:

```bash # Check disk latency ioping -c 10 /dev/sda

# Check throughput dd if=/dev/zero of=/tmp/test bs=1M count=1000 conv=fdatasync

### 5. Check context switching

Measure context switches:

```bash # System-wide context switches vmstat 1 5

# Look at 'cs' column (context switches per second) # Normal: <10,000/sec # High: >50,000/sec # Problem: >100,000/sec

# Per-process context switches pidstat -w 1 5

# Look at 'cswch/s' (voluntary) and 'nvcswch/s' (involuntary) # High voluntary = process waiting on resources # High involuntary = CPU time slice exhausted ```

Reduce context switching:

```bash # Check thread count ps -eLf | wc -l

# High thread count (>1000) causes excessive switching

# Check specific process threads ps -eLo pid,comm | grep <process> | wc -l

# Reduce thread pool sizes in application config # Common culprits: # - Java: -XX:ParallelGCThreads, executor thread pools # - Web servers: MaxWorkers, MaxConnections # - Databases: max_connections, worker_threads ```

Configure CPU affinity:

```bash # Check current CPU affinity taskset -cp <PID>

# Set CPU affinity (bind to specific cores) taskset -cp 0-3 <PID> # Bind to cores 0-3

# Or start process with affinity taskset -c 0-3 /opt/myapp/bin/myapp

# Check NUMA topology numactl --hardware

# Run process on specific NUMA node numactl --cpunodebind=0 --membind=0 /opt/myapp/bin/myapp ```

### 6. Fix cgroup CPU throttling

Check cgroup CPU limits:

```bash # For systemd services systemctl show service-name | grep -E "CPU|Quota"

# CPUQuota: CPU time percentage (100% = 1 core) # CPUWeight: Relative weight (default 100)

# Check cgroup status cat /sys/fs/cgroup/cpu/system.slice/service-name.service/cpu.stat

# Look for: # nr_throttled: Number of times cgroup was throttled # throttled_time: Total time throttled (nanoseconds) ```

Check container CPU limits:

```bash # Docker container CPU limits docker inspect <container> | grep -i cpu

# Key fields: # CpuShares: Relative weight # CpuQuota: Maximum CPU quota # CpuPeriod: Quota period (default 100ms)

# Check if container is throttled docker stats <container>

# Look at CPU % vs CPU limit

# Kubernetes pod CPU kubectl top pod <pod-name> kubectl describe pod <pod-name> | grep -A5 "Limits" ```

Adjust cgroup limits:

```bash # For systemd service # Edit /etc/systemd/system/service-name.service

[Service] CPUQuota=200% # Allow 2 CPU cores CPUWeight=150 # Higher priority than default

# Reload and restart systemctl daemon-reload systemctl restart service-name

# For Docker container docker run --cpus="2" --cpu-shares="512" ...

# Or update existing container docker update --cpus="2" <container>

# For Kubernetes, update pod spec: resources: requests: cpu: "500m" limits: cpu: "2000m" ```

### 7. Fix application-specific issues

Java application tuning:

```bash # Check Java process jps -lv

# Check GC activity jstat -gc <PID> 1000 10

# Look at: # YGC: Young GC count # YGT: Young GC time # FGC: Full GC count (should be low) # FGT: Full GC time

# High GC = memory pressure causing CPU usage

# Check heap usage jmap -heap <PID>

# Fix: Increase heap size # -Xms4g -Xmx4g (set equal to avoid resizing)

# Check thread state jstack <PID> | grep -A30 "RUNNABLE"

# Look for threads in tight loops ```

Database optimization:

```bash # MySQL: Check slow queries mysql -e "SHOW PROCESSLIST;" mysql -e "SELECT * FROM information_schema.PROCESSLIST WHERE COMMAND != 'Sleep';"

# Check for table locks mysql -e "SHOW ENGINE INNODB STATUS\G" | grep -A20 "LATEST DETECTED DEADLOCK"

# Fix: Add indexes, optimize queries, increase buffer pool # innodb_buffer_pool_size = 70% of RAM for dedicated DB server

# PostgreSQL: Check active queries psql -c "SELECT pid, usename, query, state, wait_event_type FROM pg_stat_activity WHERE state != 'idle';"

# Check for locks psql -c "SELECT * FROM pg_locks WHERE NOT granted;"

# Fix: ANALYZE tables, add indexes, increase shared_buffers ```

Web server tuning:

```bash # Nginx: Check worker configuration nginx -T | grep -E "worker_processes|worker_connections"

# Optimal: worker_processes = CPU cores # worker_connections = 1024-4096

# Apache: Check MPM configuration apache2ctl -V | grep MPM cat /etc/apache2/mods-available/mpm_prefork.conf

# Reduce MaxRequestWorkers if too high # MaxRequestWorkers = (RAM - other services) / Apache process size ```

### 8. Check for kernel issues

Check kernel scheduling:

```bash # Check scheduler statistics watch -n1 'cat /proc/schedstat'

# Look for: # sched_yield count: High = threads waiting # ttwu_count: Wakeup rate # sched_goidle: CPU going idle

# Check run queue length watch -n1 'cat /proc/loadavg'

# Load average > CPU count = scheduling delay ```

Check for interrupt storms:

```bash # Monitor interrupt rates watch -n1 'cat /proc/interrupts | awk "{sum=0; for(i=2;i<=NF;i++) sum+=\$i; print \$0, sum}"'

# Find problematic IRQ watch -n1 'cat /proc/interrupts'

# Check which device uses IRQ ls -la /sys/class/irq/<IRQ>/device/

# Disable problematic interrupt (temporary) echo 0 > /proc/irq/<IRQ>/smp_affinity_list ```

Check for soft lockups:

```bash # Enable soft lockup detection echo 1 > /proc/sys/kernel/watchdog

# Check kernel messages dmesg | grep -i "lockup\|stall"

# If soft lockups detected: # - May indicate driver bug # - May indicate hardware issue # - Update kernel or affected driver

# Check RCU (Read-Copy-Update) stalls dmesg | grep "RCU stall"

# RCU stalls indicate kernel scheduling issues ```

### 9. Implement CPU monitoring

Create monitoring script:

```bash #!/bin/bash # /usr/local/bin/cpu-monitor.sh

THRESHOLD=80 ALERT_EMAIL="admin@example.com"

# Get CPU usage CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)

# Get load average LOAD_1=$(uptime | awk -F'load average:' '{print $2}' | cut -d',' -f1 | tr -d ' ') CPU_COUNT=$(nproc)

# Check CPU usage if (( $(echo "$CPU_USAGE > $THRESHOLD" | bc -l) )); then echo "High CPU usage: ${CPU_USAGE}%" top -bn1 -o %CPU | head -20

# Send alert echo "CPU usage at ${CPU_USAGE}%" | mail -s "Alert: High CPU" $ALERT_EMAIL fi

# Check load average if (( $(echo "$LOAD_1 > $CPU_COUNT * 2" | bc -l) )); then echo "High load average: ${LOAD_1} (CPUs: $CPU_COUNT)"

# Send alert echo "Load average at ${LOAD_1}" | mail -s "Alert: High Load" $ALERT_EMAIL fi ```

Configure monitoring with systemd:

```ini # /etc/systemd/system/cpu-monitor.service [Unit] Description=CPU Usage Monitor After=network.target

[Service] Type=oneshot ExecStart=/usr/local/bin/cpu-monitor.sh

[Install] WantedBy=multi-user.target ```

```ini # /etc/systemd/system/cpu-monitor.timer [Unit] Description=Run CPU monitor every 5 minutes

[Timer] OnBootSec=5min OnUnitActiveSec=5min Unit=cpu-monitor.service

[Install] WantedBy=timers.target ```

bash # Enable monitoring systemctl daemon-reload systemctl enable cpu-monitor.timer systemctl start cpu-monitor.timer

### 10. Emergency CPU mitigation

Kill runaway process:

```bash # Graceful termination kill <PID> kill -15 <PID> # SIGTERM

# If process doesn't respond kill -9 <PID> # SIGKILL

# Kill entire process group kill -- -<PGID>

# Kill all processes by name pkill -f "process-name" killall process-name

# Kill by CPU usage (dangerous!) ps -eo pid,%cpu --sort=-%cpu | awk 'NR==2 {print $1}' | xargs kill -9 ```

Reduce process priority:

```bash # Lower process priority (nice) renice +10 -p <PID>

# Range: -20 (highest priority) to +19 (lowest) # Doesn't kill process, just reduces CPU priority

# Set I/O priority (for I/O-heavy processes) ionice -c 3 -p <PID> # Idle priority ionice -c 2 -n 7 -p <PID> # Best-effort, lowest priority ```

Emergency CPU throttling:

```bash # Limit CPU for specific process using cgroups # Create cgroup cgcreate -g cpu:/emergency_limit

# Set CPU quota (50% = half a core) echo 50000 > /sys/fs/cgroup/cpu/emergency_limit/cpu.cfs_quota_us echo 100000 > /sys/fs/cgroup/cpu/emergency_limit/cpu.cfs_period_us

# Move process to cgroup echo <PID> > /sys/fs/cgroup/cpu/emergency_limit/cgroup.procs

# Or use systemd-run for temporary limit systemd-run --scope -p CPUQuota=50% /opt/myapp/bin/myapp ```

Prevention

Set up CPU monitoring with alerting thresholds
Configure cgroup CPU limits for all services
Implement application-level rate limiting
Use connection pooling to limit concurrent requests
Profile applications before production deployment
Configure appropriate timeouts for all operations
Document known CPU-intensive operations and mitigations
Regular performance testing under load

**Load average > CPU count**: CPU contention or I/O bottleneck
**High %iowait**: Storage or network bottleneck
**High context switches**: Too many threads or processes
**CPU throttled**: Cgroup or hypervisor limits
**Soft lockup**: Kernel scheduling issue or driver bug

How to Fix Linux High CPU Usage and Load Average

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Prevention

Related Errors

Share this guide