# Docker Resource Exhausted: CPU and Memory Management

Your Docker containers are competing for resources. The host system becomes sluggish, containers slow down, or the kernel starts killing processes. Resource exhaustion in Docker isn't always obvious—containers might continue running but with degraded performance, making diagnosis challenging.

Symptoms include: - Host system becomes slow or unresponsive - Containers run but with very slow response times - Random containers die with exit code 137 (OOM) - Docker commands take a long time to execute - System shows high CPU or memory usage

Understanding Docker Resource Limits

Docker allows setting resource limits on containers:

ResourceFlagDescription
Memory--memoryHard memory limit
Memory reservation--memory-reservationSoft memory limit
CPU shares--cpu-sharesRelative CPU weight
CPU quota--cpu-quotaAbsolute CPU time limit
CPUs--cpusNumber of CPUs usable
CPU period--cpu-periodCFS scheduler period

Quick Diagnosis

Check System Resources

```bash # Overall system status top htop

# Memory usage free -h cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable"

# CPU usage mpstat 1 5

# Container resource usage docker stats --no-stream ```

Check Docker Stats

bash
docker stats

Watch for containers with: - MEM % approaching 100% - CPU % consistently high

Check System Pressure

```bash # PSI (Pressure Stall Information) - kernel 4.20+ cat /proc/pressure/memory cat /proc/pressure/cpu cat /proc/pressure/io

# Output format: some avg10=0.00 avg60=0.00 avg300=0.00 total=0 # Higher numbers indicate more pressure ```

Check cgroup Stats

bash
# For specific container
docker inspect <container> --format '{{.Id}}'
cat /sys/fs/cgroup/memory/docker/<container_id>/memory.usage_in_bytes
cat /sys/fs/cgroup/cpu/docker/<container_id>/cpu.stat

Common Causes and Fixes

Cause 1: Memory Exhaustion

Total container memory usage exceeds physical memory.

Symptoms: - Host becomes slow - Applications swap heavily - Containers killed by OOM

Diagnosis:

```bash docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Sum memory usage docker stats --no-stream --format "{{.MemUsage}}" | awk -F'/' '{sum += $1} END {print sum}' ```

Fix 1: Set memory limits on all containers

bash
docker run --memory=512m --memory-swap=1g <image>

In Docker Compose:

yaml
services:
  app:
    image: nginx
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

Fix 2: Identify and fix memory-heavy containers

```bash # Find highest memory users docker stats --no-stream | sort -k4 -h | tail -10

# Check what's inside docker exec <container> ps aux --sort=-%mem docker exec <container> top -o %MEM ```

Fix 3: Restart problematic containers

bash
docker restart <memory-hogging-container>

Fix 4: Add physical memory or enable swap

```bash # Check swap free -h | grep Swap

# Add swap file sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile ```

Cause 2: CPU Exhaustion

Containers collectively using all CPU capacity.

Symptoms: - Slow response times - High CPU % in docker stats - Host processes slow

Diagnosis:

```bash docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}"

# Check CPU saturation mpstat 1 5 # Look at %idle - low values mean CPU saturated ```

Fix 1: Set CPU limits

```bash # Limit to 1 CPU docker run --cpus=1 <image>

# Limit with quota/period (more precise) docker run --cpu-period=100000 --cpu-quota=50000 <image> # 0.5 CPUs ```

In Docker Compose:

yaml
services:
  app:
    image: nginx
    deploy:
      resources:
        limits:
          cpus: '0.5'

Fix 2: Use CPU shares for priority

bash
# Default is 1024
docker run --cpu-shares=512 <image>  # Lower priority
docker run --cpu-shares=2048 <image>  # Higher priority

Fix 3: Identify CPU-heavy processes

bash
docker exec <container> ps aux --sort=-%cpu
docker exec <container> top -o %CPU

Fix 4: Stop unnecessary containers

bash
docker stop <cpu-intensive-container>

Cause 3: Disk I/O Exhaustion

Containers performing heavy disk operations.

Symptoms: - Slow container startup - Slow Docker commands - High iowait in top

Diagnosis:

```bash # Check I/O wait top | head -5 # Look for high wa (iowait)

# Check disk stats iostat -x 5 5

# Check container disk usage docker ps -as ```

Fix 1: Limit disk I/O

```bash # Limit read rate docker run --device-read-bps=/dev/sda:10mb <image>

# Limit write rate docker run --device-write-bps=/dev/sda:10mb <image>

# Limit IOPS docker run --device-read-iops=/dev/sda:1000 <image> ```

Fix 2: Use volumes instead of container layers

bash
# Volume I/O is faster
docker run -v data:/app/data <image>

Fix 3: Use tmpfs for temporary data

bash
docker run --tmpfs /tmp:rw,size=100m <image>

Fix 4: Distribute across disks

bash
# Move containers to different disks
docker run -v /fast-ssd/data:/app/data <image>
docker run -v /hdd/logs:/app/logs <image>

Cause 4: Network Resource Exhaustion

Excessive network connections or bandwidth.

Symptoms: - Network timeouts - Slow network operations - Connection failures

Diagnosis:

```bash # Check network connections ss -s netstat -s

# Check bandwidth iftop nethogs ```

Fix 1: Limit connections

bash
# Use connection pooling in applications
# Limit worker processes in containers

Fix 2: Distribute load

yaml
# Use multiple containers instead of one heavy one
services:
  worker:
    image: myimage
    scale: 4
    deploy:
      resources:
        limits:
          memory: 256M

Cause 5: Too Many Containers

Running more containers than system can handle.

Symptoms: - Docker commands slow - Container startup slow - System unresponsive

Diagnosis:

bash
docker ps | wc -l
docker ps -q | wc -l

Fix: Reduce running containers

```bash # Stop unused containers docker stop $(docker ps -q --filter "status=running")

# Remove stopped containers docker container prune ```

Cause 6: Resource Limits Not Effective

Limits set but not preventing exhaustion.

Symptoms: - Limits set but host still exhausted - OOM kills despite memory limits

Fix 1: Ensure limits are enforced

bash
# Check if limits are applied
docker inspect <container> --format '{{json .HostConfig.Memory}}'
cat /sys/fs/cgroup/memory/docker/<id>/memory.limit_in_bytes

Fix 2: Disable swap for strict limits

bash
docker run --memory=512m --memory-swap=512m <image>
# memory-swap = memory means no swap allowed

Fix 3: Update limits for running containers

Docker can't update limits on running containers. You must recreate:

bash
docker stop <container>
docker rm <container>
docker run --memory=1g <image> --name <container>

Setting Appropriate Limits

Memory Limit Strategy

  1. 1.Measure actual usage
  2. 2.Set limit 20-30% above peak
  3. 3.Set reservation at average usage

```bash # Monitor usage first docker stats <container> --no-stream

# Then set limits docker run --memory=1g --memory-reservation=512m <image> ```

CPU Limit Strategy

  1. 1.Check actual CPU usage
  2. 2.Set limit based on needed performance
  3. 3.Use shares for relative priority
bash
# If container uses 50% of one CPU consistently
docker run --cpus=0.75 <image>  # Give it 75% to have headroom

Docker Compose Resource Management

```yaml services: critical-app: image: nginx deploy: resources: limits: cpus: '2' memory: 2G reservations: cpus: '1' memory: 1G

background-worker: image: worker deploy: resources: limits: cpus: '0.5' memory: 256M reservations: cpus: '0.1' memory: 128M ```

Monitoring Resource Usage

Continuous Monitoring

```bash # Watch stats watch -n 5 docker stats --no-stream

# Log stats docker stats --no-stream >> /var/log/docker-stats.log & ```

Alert Script

```bash #!/bin/bash # alert-resources.sh THRESHOLD=90

for container in $(docker ps -q); do mem=$(docker stats --no-stream --format "{{.MemPerc}}" $container | tr -d '%') if (( mem > THRESHOLD )); then name=$(docker inspect $container --format '{{.Name}}') echo "WARNING: $name memory at ${mem}%" fi done ```

Prometheus/Grafana Integration

Use cAdvisor for detailed monitoring:

bash
docker run -d --name=cadvisor \
  -p 8080:8080 \
  -v /:/rootfs:ro \
  -v /var/run:/var/run:ro \
  -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  google/cadvisor:latest

Verification Steps

  1. 1.Verify limits applied:
  2. 2.```bash
  3. 3.docker inspect <container> --format '{{.HostConfig.Memory}}'
  4. 4.docker inspect <container> --format '{{.HostConfig.CpuQuota}}'
  5. 5.`
  6. 6.Check resource usage decreased:
  7. 7.```bash
  8. 8.docker stats --no-stream <container>
  9. 9.`
  10. 10.Test container performance:
  11. 11.```bash
  12. 12.docker exec <container> curl http://localhost/health
  13. 13.`
  14. 14.Verify system resources:
  15. 15.```bash
  16. 16.free -h
  17. 17.top -bn1 | head -10
  18. 18.`
  19. 19.Test under load:
  20. 20.```bash
  21. 21.# Stress test
  22. 22.docker exec <container> stress --vm 1 --vm-bytes 100M
  23. 23.# Should not exceed memory limit
  24. 24.`

Resource exhaustion requires a holistic approach—set appropriate limits on all containers, monitor aggregate usage, and ensure total allocated resources don't exceed physical capacity. The key is balancing container needs with system limits.