What's Actually Happening

Your Docker container is being killed by the OOM (Out of Memory) killer because it's exceeding its allocated memory limit, or the host system is running out of memory. Containers exit with code 137 (128 + 9 = SIGKILL) indicating they were forcibly terminated.

The Error You'll See

```bash $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES abc123def456 my-app "node app" 2 minutes ago Exited (137) 1 minute ago my-container

$ docker logs my-container # Logs may end abruptly with no error message

$ docker inspect my-container --format='{{.State.OOMKilled}}' true

$ docker events 2024-01-01T12:00:00.000000000Z container die abc123def456 (exitCode=137, oomKilled=true)

$ docker-compose up ERROR: Container exited with code 137 ```

Why This Happens

  1. 1.Container memory limit too low: Application needs more memory than allocated
  2. 2.Memory leak: Application leaking memory over time
  3. 3.Heap size too large: JVM or Node.js heap exceeds container memory
  4. 4.No memory limit set: Container uses unlimited memory until host OOM
  5. 5.Swap disabled: No swap space available when memory is exhausted
  6. 6.Memory fragmentation: Cannot allocate even with free memory
  7. 7.Multiple containers: Combined usage exceeds host memory
  8. 8.Background processes: Unexpected processes consuming memory

Step 1: Identify OOM Kills

Confirm the container was OOM killed:

```bash # Check container exit code docker inspect <container-id> --format='{{.State.ExitCode}}' # 137 = OOM killed (128 + 9 SIGKILL)

# Check if OOM killed specifically docker inspect <container-id> --format='{{.State.OOMKilled}}' # true = OOM killed

# Check container memory limit docker inspect <container-id> --format='{{.HostConfig.Memory}}'

# Check memory usage history docker stats <container-id> --no-stream

# View container events docker events --filter container=<container-id> --since 1h

# Check system OOM logs dmesg | grep -i "killed process" journalctl -g "Out of memory" ```

Step 2: Check Current Memory Usage

Analyze memory usage:

```bash # Check host memory free -h

# Check Docker memory usage docker system df

# Live memory stats for all containers docker stats

# Memory stats for specific container docker stats <container-id> --no-stream

# Detailed memory info cat /sys/fs/cgroup/memory/docker/<container-id>/memory.usage_in_bytes cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes cat /sys/fs/cgroup/memory/docker/<container-id>/memory.max_usage_in_bytes

# Check memory breakdown in container docker exec <container-id> cat /proc/meminfo ```

Step 3: Increase Container Memory Limit

Set appropriate memory limits:

```bash # Run with memory limit docker run --memory="1g" --memory-swap="2g" my-image

# Run with memory limit and swap docker run --memory="1g" --memory-swap="2g" my-image

# Disable swap for container docker run --memory="1g" --memory-swap="1g" my-image

# Run with memory reservation (soft limit) docker run --memory="2g" --memory-reservation="1g" my-image ```

In docker-compose.yml: ``yaml version: '3.8' services: app: image: my-image deploy: resources: limits: memory: 2G reservations: memory: 1G

Or using older compose format: ``yaml version: '3.8' services: app: image: my-image mem_limit: 2g memswap_limit: 4g mem_reservation: 1g

Step 4: Configure Swap Memory

Enable swap to prevent OOM kills:

```bash # Check if swap is enabled swapon --show

# Enable swap if not available sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile

# Make swap permanent echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Configure swappiness sudo sysctl vm.swappiness=10 echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf ```

Configure Docker to use swap: ```bash # Run container with swap docker run --memory="1g" --memory-swap="2g" my-image # memory-swap = memory + swap, so swap = 1g

# Disable OOM killer for container (use with caution) docker run --oom-kill-disable --memory="1g" my-image

# Set OOM score adjustment docker run --oom-score-adj=500 my-image # Range: -1000 (never kill) to 1000 (kill first) ```

Step 5: Tune Application Memory Settings

Configure application memory usage:

For Java applications: ```bash # Set JVM heap size to fit within container memory docker run -e JAVA_OPTS="-Xms512m -Xmx1536m" --memory="2g" my-java-app

# Use container-aware JVM (Java 10+) docker run -e JAVA_OPTS="-XX:MaxRAMPercentage=75.0" --memory="2g" my-java-app

# In Dockerfile ENV JAVA_OPTS="-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0" ```

For Node.js applications: ```bash # Set Node.js heap size docker run -e NODE_OPTIONS="--max-old-space-size=1536" --memory="2g" my-node-app

# In Dockerfile ENV NODE_OPTIONS="--max-old-space-size=1536"

# Or in code # const v8 = require('v8'); # const heapStats = v8.getHeapStatistics(); ```

For Python applications: ```bash # Limit Python memory (not directly possible, use resource limits) docker run --memory="1g" --memory-swap="1g" my-python-app

# Use memory profiler pip install memory-profiler python -m memory_profiler app.py ```

Step 6: Enable Swap for Container

Configure swap at container level:

bash
# In daemon.json
sudo nano /etc/docker/daemon.json
json
{
  "memory-swap": true
}

```bash # Restart Docker sudo systemctl restart docker

# Run with swap enabled docker run --memory="1g" --memory-swap="2g" my-image ```

Step 7: Debug Memory Usage

Profile memory usage inside container:

```bash # Enter container docker exec -it <container-id> sh

# Inside container, check processes ps aux --sort=-%mem

# Check memory info cat /proc/meminfo

# Check specific process memory cat /proc/<pid>/status | grep -i mem

# Check memory maps pmap -x <pid>

# For Java apps jmap -heap <pid> jmap -histo <pid> | head -20

# For Node.js apps node --inspect app.js # Then connect Chrome DevTools

# Use Docker stats continuously docker stats <container-id> ```

Step 8: Optimize Memory Usage

Reduce memory footprint:

```dockerfile # Use smaller base images FROM alpine:3.18 # Instead of ubuntu:latest

# Use multi-stage builds FROM node:18-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm ci --only=production

FROM node:18-alpine WORKDIR /app COPY --from=builder /app/node_modules ./node_modules COPY . . CMD ["node", "app.js"]

# Disable unnecessary features ENV NODE_ENV=production ENV DEBUG=false

# Clean up in same layer RUN apt-get update && apt-get install -y package \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* ```

Step 9: Monitor Memory with Alerts

Set up memory monitoring:

```bash # Create a monitoring script cat > monitor-memory.sh << 'EOF' #!/bin/bash CONTAINER=$1 LIMIT=$(docker inspect $CONTAINER --format='{{.HostConfig.Memory}}') THRESHOLD=$((LIMIT * 90 / 100))

while true; do USAGE=$(docker stats $CONTAINER --no-stream --format '{{.MemUsage}}') USED=$(docker stats $CONTAINER --no-stream --format '{{.MemPerc}}' | tr -d '%')

if (( $(echo "$USED > 90" | bc -l) )); then echo "WARNING: Container $CONTAINER using $USED% memory" # Send alert, restart container, etc. fi

sleep 10 done EOF

chmod +x monitor-memory.sh ./monitor-memory.sh my-container ```

Using Prometheus for monitoring: ```yaml # prometheus.yml global: scrape_interval: 15s

scrape_configs: - job_name: 'cadvisor' static_configs: - targets: ['cadvisor:8080']

# docker-compose.yml services: cadvisor: image: gcr.io/cadvisor/cadvisor:latest volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro ports: - "8080:8080" ```

Step 10: Use Docker Memory Features

Leverage Docker's memory management:

```bash # Set memory swappiness (how aggressive to use swap) docker run --memory="1g" --memory-swappiness=50 my-image

# Set kernel memory limit (TCP buffers, etc.) docker run --memory="1g" --kernel-memory="500m" my-image

# Memory pressure handling docker run --memory="1g" --memory-reservation="512m" my-image

# Disable OOM killer (container won't be killed, but may become unresponsive) docker run --memory="1g" --oom-kill-disable my-image

# Adjust OOM score (higher = more likely to be killed) docker run --memory="1g" --oom-score-adj=500 my-image ```

Verify the Fix

Confirm containers run without OOM kills:

```bash # Run container with proper memory limit docker run -d --name test-app --memory="2g" --memory-swap="4g" my-image

# Monitor memory usage docker stats test-app --no-stream

# Watch for OOM in events docker events --filter container=test-app &

# Check after running for a while docker inspect test-app --format='{{.State.OOMKilled}}' # Should return: false

# Check exit code when container stops docker inspect test-app --format='{{.State.ExitCode}}' # Should not be 137

# Verify container is still running docker ps | grep test-app ```

Your container should now run without being OOM killed.