1. What Does OOMKilled (Exit Code 137) Mean?

In Kubernetes, an OOMKilled status occurs when the Linux kernel's Out-Of-Memory (OOM) killer forcibly terminates a containerized process because it exceeded its allocated memory threshold.

When a process is killed by the OOM killer, it receives the fatal signal SIGKILL (Signal 9). The container runtime reports this as Exit Code 137, derived from standard POSIX convention: 128 + 9 = 137.

Container Memory Limit vs. Node Memory Pressure

There are two distinct scenarios where OOM events occur:

Container cgroup limit exceeded: The container consumed more RAM than specified in resources.limits.memory. The node has plenty of free memory, but the container was killed by its own cgroup limit.
Node-level memory exhaustion: The worker node itself ran out of allocatable RAM. In this case, the Kubelet triggers pod eviction based on QoS classes (BestEffort first, then Burstable).

2. How to Recognize and Confirm OOMKilled

To reliably verify whether an unexpected container restart was caused by OOMKilled, inspect the terminated container state using kubectl:

Query container termination state and exit code

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{": ExitCode="}{.lastState.terminated.exitCode}{" Reason="}{.lastState.terminated.reason}{"\n"}{end}'

Illustrative output:

api-worker: ExitCode=137 Reason=OOMKilled

Next, inspect the pod description to view resource limits and historical termination timestamps:

Inspect pod limits and last terminated state

kubectl describe pod <pod-name> -n <namespace>

Look specifically for the Last State block under Containers:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 16 Oct 2026 14:10:00 +0000
      Finished:     Fri, 16 Oct 2026 14:18:22 +0000

3. Inspecting Memory Usage: Metrics & Leak Analysis

Before taking action, determine whether the container experienced a spike under legitimate traffic load or a steady, unbounded memory leak:

Step 3.1: Check Live Container Memory Consumption

Query live memory consumption across pods

kubectl top pod <pod-name> -n <namespace> --containers

Step 3.2: Distinguish Between Genuine Growth and Memory Leaks

Legitimate Spike: Memory usage climbs during high concurrency, batch jobs, or large payload processing, and returns to baseline when traffic subsides.
Memory Leak: Memory usage climbs monotonically over hours or days regardless of request volume until hitting the cgroup limit and restarting.
JVM / Runtime Heap Misconfiguration: In runtimes such as Java (JVM) or Node.js, the runtime heap (-Xmx or --max-old-space-size) is set equal to or higher than the container cgroup limit, leaving no margin for native buffers, thread stacks, or OS cache.

4. Why Blindly Increasing Limits Can Make It Worse

The Pitfall of Premature Limit Increases

When a container crashes with Exit Code 137, an operator's first instinct is often to double the memory limit (e.g. from 512Mi to 1Gi). While this may delay restarts, it frequently creates severe cascading cluster risks:

Prolonging Memory Leaks: If your application has a memory leak, increasing limits only increases the time between crashes; it does not solve the failure.
Node Compaction & Eviction: If multiple replicas have inflated limits, an unexpected load surge can trigger node-level Out-of-Memory, resulting in Kubelet evicting critical cluster components.
Longer GC Pauses: For garbage-collected runtimes like Java, doubling the heap can exponentially increase stop-the-world garbage collection pause times, causing failed liveness probes.

5. Safe Remediation Options and Trade-offs

Choose the appropriate remediation strategy based on your diagnosis:

Option 1: Legitimate Limit Adjustment (When Requests Equal Limits)

If profiling confirms the application legitimately requires more memory under standard production workloads, patch the deployment limit safely:

Patch Deployment memory limit safely

kubectl patch deployment <deployment-name> -n <namespace> \
  --type='json' -p='[
    {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "1Gi"},
    {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "512Mi"}
  ]'

Option 2: Align JVM / Runtime Heap Flags

Always ensure runtime maximum heap memory is set to approximately 70-75% of the container cgroup limit, reserving 25-30% for native memory and thread stacks:

Example JVM Container Support Flag

JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0

Option 3: How AetherKube Prevents Cascading OOM Restarts

AetherKube detects OOM spikes before the kernel kills the cgroup. Under controlled autonomous healing policies, AetherKube can apply proportional burst buffers, notify on-call teams with exact stack profiling, and enforce QoS class boundaries to prevent node-level evictions.

6. Verification and Long-term Prevention

Verify that the fix has stabilized memory usage without causing node starvation:

Monitor container memory consumption over time

kubectl top pod -l app=<app-label> -n <namespace>

Long-term Prevention Recommendations:

Always set both Requests and Limits: Setting memory requests guarantees scheduled headroom; setting limits prevents a runaway container from degrading neighbor pods.
Configure Prometheus Alerts on Memory Saturation: Trigger warning alerts when container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85 for more than 5 minutes.
Leverage Vertical Pod Autoscaler (VPA) in Recommendation Mode: Use VPA to analyze real historical memory usage patterns without automatically restarting production pods.

Frequently Asked Questions

What is the difference between working set memory and total memory?

Kubernetes uses container_memory_working_set_bytes (not total resident memory) to determine if a container exceeds its memory limit. Working set memory excludes inactive file cache that the operating system can freely reclaim.

Where can I read official Kubernetes resource management guidelines?

Consult the official Kubernetes documentation on Managing Resources for Containers .

Prevent Container OOM Crashes with AetherKube

Download AetherKube v1.0.0 Public Beta to monitor cluster memory thresholds and execute controlled self-healing.

Get Public Beta