Introduction

Kubernetes OOMKilled occurs when a container exceeds its memory limit and is terminated by the Linux OOM killer. The pod shows Exit Code: 137 (128 + SIGKILL=9) and Reason: OOMKilled in the container status. This is different from application-level OutOfMemoryError - the kernel's OOM killer terminates the process when cgroup memory limit is exceeded. OOMKilled pods cause service disruptions, data loss, and cascading failures if not addressed. Common causes include memory leaks, insufficient limits, traffic spikes, or incorrect memory accounting.

Symptoms

  • kubectl get pods shows OOMKilled or CrashLoopBackOff
  • Container exit code 137 (SIGKILL from OOM killer)
  • kubectl describe pod shows Reason: OOMKilled in Last State
  • Pod restarts frequently with increasing memory usage before each crash
  • Issue appears after traffic increase, new deploy, or node memory pressure
  • Application may not have time to log OOM errors before being killed
  • Node shows dmesg entries about OOM killer invocation

Common Causes

  • Container memory limit set too low for application requirements
  • Memory leak in application causing unbounded growth
  • Traffic spike causing temporary memory surge
  • Java application not respecting container memory limits
  • Init container OOMKilled preventing pod startup
  • Node-level memory pressure triggering OOM killer
  • Memory limit lower than application working set
  • Sidecar containers consuming unexpected memory

Step-by-Step Fix

### 1. Confirm OOMKilled diagnosis

Verify the termination reason:

```bash # Check pod status kubectl get pod <pod-name> -o wide

# Output: # NAME READY STATUS RESTARTS AGE # myapp 0/1 CrashLoopBackOff 15 1h

# Describe pod for detailed status kubectl describe pod <pod-name>

# Look for: # Last State: Terminated # Reason: OOMKilled # Exit Code: 137

# Or use jsonpath for quick check kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}' # Expected output: OOMKilled

# Check exit code kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.exitCode}' # Expected output: 137

# Check all pods in namespace for OOMKilled kubectl get pods -n <namespace> -o json | \ jq -r '.items[] | select(.status.containerStatuses[].lastState.terminated.reason == "OOMKilled") | .metadata.name' ```

Check OOM kill history:

```bash # Kubernetes events may show OOM kills kubectl get events --field-selector reason=OOMKilled -n <namespace>

# Check previous pod instances kubectl logs <pod-name> -n <namespace> --previous

# If container was OOMKilled, logs may be truncated or missing ```

### 2. Check container memory limits

Verify resource configuration:

```bash # Get container resource limits kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources}'

# Or in yaml format kubectl get pod <pod-name> -o yaml | grep -A10 "resources:"

# Expected output: # resources: # limits: # memory: 512Mi # cpu: "500m" # requests: # memory: 256Mi # cpu: "250m"

# Check limits for all containers in pod (including sidecars) kubectl get pod <pod-name> -o json | \ jq '.spec.containers[] | {name: .name, memory_limit: .resources.limits.memory}'

# Common issue: Sidecar with low limits # Main app: 512Mi, Sidecar: 128Mi (may be too low) ```

Check LimitRange and ResourceQuota:

```bash # Check LimitRange in namespace (default limits) kubectl get limitrange -n <namespace> -o yaml

# Output: # spec: # limits: # - type: Container # default: # memory: 512Mi # cpu: "500m" # defaultRequest: # memory: 256Mi # cpu: "250m"

# Check ResourceQuota kubectl get resourcequota -n <namespace> -o yaml

# If quota is tight, pods may get lower limits than needed ```

### 3. Analyze memory usage patterns

Get historical memory usage:

```bash # If metrics-server is installed kubectl top pod <pod-name> -n <namespace>

# Output: # NAME CPU(cores) MEMORY(bytes) # myapp 50m 480Mi

# If memory is close to or exceeding limit, OOM is likely # MEMORY should be < 80% of limit for safety margin

# Get memory usage over time (requires Prometheus) kubectl run -it --rm --restart=Never debug --image=alpine -- \ curl -s "http://prometheus:9090/api/v1/query_range?query=container_memory_usage_bytes{pod=\"<pod-name>\"}&start=$(date -d '1 hour ago' -Iseconds)&end=$(date -Iseconds)&step=1m"

# Or use kubectl plugin kubectl prometheus pod <pod-name> --duration 1h ```

Check memory usage inside container:

```bash # Exec into container (if it stays running long enough) kubectl exec -it <pod-name> -n <namespace> -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# For cgroup v2 kubectl exec -it <pod-name> -n <namespace> -- cat /sys/fs/cgroup/memory.max

# Check current usage kubectl exec -it <pod-name> -n <namespace> -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes

# Check memory stats kubectl exec -it <pod-name> -n <namespace> -- cat /sys/fs/cgroup/memory/memory.stat

# Key metrics: # cache: Page cache (can be reclaimed) # rss: Resident Set Size (actual memory used by process) # mapped_file: Memory-mapped files # inactive_file: Reclaimable file cache ```

### 4. Check for memory leaks

Identify if application has memory leak:

```bash # Monitor memory growth over time # Run this in a loop to see growth pattern watch -n 5 'kubectl top pod <pod-name> -n <namespace>'

# Memory leak pattern: # - Memory grows steadily even under constant load # - Memory doesn't decrease after traffic spike # - Growth continues until OOMKilled

# Healthy pattern: # - Memory stable under constant load # - Memory spikes during traffic then returns to baseline # - GC effectively reclaims memory

# Check restart frequency kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].restartCount}'

# If restart count increases every few minutes/hours: # - Memory leak likely # - Or limit too low for workload ```

Application-level leak detection:

```bash # For Java applications kubectl exec -it <pod-name> -n <namespace> -- \ jcmd 1 GC.heap_info

# For .NET applications kubectl exec -it <pod-name> -n <namespace> -- \ dotnet-dump collect --output /tmp/dump

# For Go applications (with pprof) kubectl port-forward <pod-name> 6060:6060 & curl http://localhost:6060/debug/pprof/heap > heap.prof go tool pprof heap.prof

# Check for goroutine leaks curl http://localhost:6060/debug/pprof/goroutine?debug=2 ```

### 5. Adjust memory limits appropriately

Set limits based on actual usage:

```bash # Rule of thumb: Limit = 1.5-2x normal usage # This allows headroom for spikes without OOM

# Example: If normal usage is 400Mi # Set limit to 600Mi-800Mi

# Update deployment kubectl edit deployment <deployment-name> -n <namespace>

# Update resources section: spec: template: spec: containers: - name: <container-name> resources: limits: memory: 768Mi # Increased from 512Mi cpu: "500m" requests: memory: 384Mi # Also increase request cpu: "250m"

# Or use kubectl patch kubectl patch deployment <deployment-name> -n <namespace> --type=json -p='[ {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "768Mi"}, {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "384Mi"} ]' ```

Memory limit guidelines by workload:

Workload Type Normal Usage Recommended Limit ───────────────────────────────────────────────────────────── Java Spring Boot 500-800Mi 1-1.5Gi Node.js API 100-200Mi 256-512Mi Python Flask 150-300Mi 384-512Mi Go Microservice 50-100Mi 128-256Mi Redis Cache 200-500Mi 512Mi-1Gi Nginx Proxy 20-50Mi 64-128Mi Sidecar (Envoy) 100-200Mi 256-512Mi Init Container 50-100Mi 128-256Mi

### 6. Configure Java for container memory

Java applications need special configuration:

```yaml # WRONG: Java not aware of container limits # Default JVM heap may exceed container limit # Causes immediate OOMKilled

resources: limits: memory: 512Mi # JVM sees full node memory, sets heap to 1-2GB # Container gets killed when exceeding 512Mi

# CORRECT: Configure JVM for container spec: containers: - name: java-app resources: limits: memory: 512Mi env: - name: JAVA_OPTS # Java 8u191+ automatically detects container limits # For older Java, explicitly set heap value: "-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0" # 75% of 512Mi = 384Mi heap, leaving room for non-heap

# Or via command line args: - -XX:MaxRAMPercentage=75.0 - -XX:InitialRAMPercentage=50.0 - -XX:+UseContainerSupport # Enabled by default in Java 10+ - -XX:MaxRAM=512m # Explicit max for older Java - -Xms256m - -Xmx384m

# For Java 11+ with G1GC args: - -XX:MaxRAMPercentage=75.0 - -XX:+UseG1GC - -XX:MaxGCPauseMillis=200 - -XX:+UseContainerSupport ```

Java memory breakdown in containers:

``` Container Memory Limit: 512Mi ├── Heap (XMX): 384Mi (75%) ├── Metaspace: 64Mi ├── Code Cache: 32Mi ├── Thread Stacks: 16Mi (256 threads × 64KB) ├── Direct Buffers: 8Mi └── GC Structures: 8Mi ───────────────────────────────── Total: 512Mi

# If Non-heap > expected, reduce heap percentage # Try MaxRAMPercentage=65.0 instead of 75.0 ```

### 7. Check node-level OOM pressure

Node memory pressure can kill pods:

```bash # Check node memory status kubectl describe node <node-name>

# Look for: # Conditions: # Type: MemoryPressure # Status: True # If True, node is under memory pressure

# Check allocatable memory kubectl describe node <node-name> | grep -A5 "Allocatable:"

# Output: # Allocatable: # cpu: 4 # ephemeral-storage: 100Gi # hugepages-1Gi: 0 # hugepages-2Mi: 0 # memory: 7675Mi # This is total allocatable

# Check what's using memory kubectl top nodes

# If node memory > 90%, OOM killer may target pods ```

Check system OOM kills:

```bash # SSH to the node where pod was running # Check kernel OOM messages dmesg -T | grep -i "oom\|killed"

# Expected output: # [Mar30 10:15] Out of memory: Killed process 12345 (java) total-vm:2048000kB # This shows kernel OOM killer invoked

# Check cgroup OOM events cat /sys/fs/cgroup/memory/memory.oom_control

# For cgroup v2 cat /sys/fs/cgroup/memory.stat | grep oom

# If oom_kill > 0, OOM killer was invoked ```

Adjust pod priority to avoid being killed:

yaml # Higher priority pods are killed last apiVersion: v1 kind: Pod metadata: name: critical-app spec: priorityClassName: high-priority # Must exist in cluster containers: - name: app # ... --- # Create PriorityClass if not exists apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 # Higher = more important globalDefault: false description: "Critical applications that should not be OOM killed"

### 8. Implement memory monitoring and alerting

Set up proactive monitoring:

```yaml # Prometheus recording rules groups: - name: kubernetes_memory rules: - record: pod:memory_usage_ratio expr: | container_memory_usage_bytes / container_spec_memory_limit_bytes

  • record: pod:memory_rss_bytes
  • expr: container_memory_rss

# Alerting rules groups: - name: kubernetes_oom rules: - alert: PodMemoryHigh expr: pod:memory_usage_ratio > 0.85 for: 10m labels: severity: warning annotations: summary: "Pod {{ $labels.pod }} memory above 85%" description: "Memory usage is {{ $value | humanizePercentage }}"

  • alert: PodOOMKilled
  • expr: increase(kube_pod_container_status_restarts_total[1h]) > 2
  • for: 5m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Pod {{ $labels.pod }} restarting frequently"
  • description: "Possible OOMKilled - {{ $value }} restarts in last hour"
  • alert: NodeMemoryPressure
  • expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Node {{ $labels.node }} under memory pressure"
  • `

### 9. Handle init container OOMKilled

Init containers can also OOM:

```bash # Check init container status kubectl describe pod <pod-name> | grep -A10 "Init Containers:"

# If init container shows OOMKilled: # Init Container 1: # State: Terminated # Reason: OOMKilled # Exit Code: 137

# Init containers share node memory with main containers # But have separate limits ```

Fix init container OOM:

```yaml spec: initContainers: - name: init-migration image: myapp/migration:latest resources: limits: memory: 256Mi # Increase from default cpu: "500m" requests: memory: 128Mi cpu: "250m" command: ["sh", "-c", "run-migration"]

containers: - name: main-app # ... ```

### 10. Implement graceful degradation

Handle memory pressure gracefully:

yaml # Use ephemeral storage for temporary data # Prevents memory accumulation apiVersion: v1 kind: Pod metadata: name: myapp spec: containers: - name: app volumeMounts: - name: tmp mountPath: /tmp - name: cache mountPath: /var/cache volumes: - name: tmp emptyDir: sizeLimit: 1Gi # Limit tmp space - name: cache emptyDir: sizeLimit: 512Mi # Limit cache space

Configure Liveness probe with memory awareness:

```yaml spec: containers: - name: app livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 3 # Pod will be restarted if health check fails # This is better than waiting for OOMKilled

readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5 # Remove from service if not ready failureThreshold: 3 ```

Prevention

  • Set memory limits to 1.5-2x normal usage based on load testing
  • Configure Java with MaxRAMPercentage for containers
  • Monitor memory usage continuously with Prometheus
  • Set up alerts for memory usage > 85%
  • Use Volumes for temporary/cache data instead of memory
  • Test memory limits in staging before production
  • Document memory requirements for each service
  • Review memory limits after each deploy
  • **CrashLoopBackOff**: Pod repeatedly crashing (may be OOMKilled)
  • **Evicted**: Pod removed due to resource pressure
  • **Pending**: Pod cannot be scheduled (insufficient resources)
  • **NodeNotReady**: Node unavailable (may trigger OOM on remaining nodes)