Kubernetes OOMKilled and CPU throttling: diagnose requests, limits, and HPA

October 25, 2026 · 10 min read

Investigate restarting Pods, latency despite low average CPU, and ineffective scaling. For Linux containers, connect resource settings, OOM versus eviction, HPA's utilization denominator, and the database connection budget.

How requests differ from limits

This article covers per-container CPU and memory settings on Linux nodes. Pod-level resources, MemoryQoS, and runtimes can affect behavior, so inspect the settings actually applied. The following values illustrate the model; they are not service recommendations.

Container excerpt: m is one thousandth of a CPU core; Mi means mebibytes

# Excerpt from one container's spec; hypothetical values, not defaults
resources:
  requests:
    cpu: "200m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Setting	Primary role	Common misunderstanding
CPU requests	Input to placement and relative allocation during CPU contention	A 200m request does not cap usage at 200m
CPU limits	Constrain the allocation of CPU time	500m does not mean exactly half a core at every instant
Memory requests	Memory requirement used for placement	The requested amount is not memory already dedicated to the application
Memory limits	Constrain memory; pressure can result in OOM termination	Crossing the configured value does not necessarily cause an immediate kill

Low current memory usage on a node does not let the scheduler ignore a new Pod's requests. Conversely, lowering requests to make a Pod fit does not reduce its actual memory needs. Examine placement density and runtime headroom separately.

Collect termination reasons and preceding conditions first

Inspect Pod status, the affected container's last termination reason, restart count, events, and previous-run logs. These are investigation examples, not results from a live cluster. The top command shows current samples; use historical monitoring to capture a peak before termination.

kubectl: example read-only status and log commands

# Replace namespace, Pod, container, and HPA names for your environment
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -c <container> -n <namespace> --previous
kubectl top pods -n <namespace> --containers
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp
kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>

Symptom	Evidence	Next step
Container restarts	lastState reason, previous logs, memory history, node state	Separate OOM, application exits, and probe failures
Pod becomes Evicted	Pod reason/message, node MemoryPressure or disk state	Investigate node headroom and eviction events
Slow without restarting	Request latency, CPU limits, cgroup throttling statistics	Compare CPU constraints with database and lock waits
HPA requests more replicas but Pods stay Pending	FailedScheduling events and resource requests	Inspect node capacity, placement constraints, and quotas

Separate OOMKilled from node eviction

OOMKilled suggests memory-related termination, but investigate whether the container limit or a node-wide OOM was involved. Exit code 137 alone is insufficient: other SIGKILL terminations can produce it. Correlate the reason, node records, and memory history. Kubelet eviction after detecting node pressure is another path, distinct from a container's OOM termination.

Heap size does not explain all memory. Include native allocations, threads, buffers, caches, and file-related memory represented in the measurement. Whether usage follows concurrency or batch size, grows after traffic subsides, or peaks only at startup changes the choice between adjusting limits, bounding concurrency, and fixing a leak.

Before raising a limit, check simultaneous node-wide peaks and placement headroom. Avoid merely transferring pressure to other Pods.
Equal requests and limits do not fix an application needing more than its limit. A QoS classification alone does not prevent OOM.
Leave room between runtime heap settings and the container limit for non-heap memory and variation. Determine headroom through measurement.

Why low average CPU does not rule out throttling

Linux CPU bandwidth control can make execution wait for replenishment after a period's CPU-time budget is exhausted. Short parallel bursts separated by idle time can combine low long-window average CPU with short-window constraints and latency. Compare latency and increases in throttling over the same interval instead of ruling out CPU constraints from the average.

More throttling events alone do not establish the cause of worse p99 latency. Check database waits, garbage collection, locks, and node contention, then compare limits or concurrency under the same traffic and workload. Removing this container's CPU limit can leave ancestor cgroup limits and CPU contention in place. Check the runtime version and its container-awareness before changing parallelism settings.

HPA CPU utilization divides by requests, not limits

Consider HPA using a CPU Resource metric with a Utilization target. A single-container Pod using 100m with a 200m request has 50% utilization. Lowering the request to 100m makes the same usage 100%. A request change affects both placement and scaling decisions. Missing relevant requests on containers can also leave the metric undefined.

With four Pods averaging 100% against a 50% target, the basic formula ceil(4 * 100 / 50) produces eight Pods. This simplifies to valid metrics for every Pod; tolerance, unready Pods, missing metrics, min/maxReplicas, and scaling policies affect the actual decision. A desired count of eight does not mean eight workers are immediately serving traffic.

From a larger desired replica count to useful capacity: measure delays at each stage.

HPA updates desired count → Obtain a schedulable node → Pull image and start process → Initialize and pass readiness → New Pod joins processing

For Pending Pods, inspect scheduling events. For running but unready Pods, inspect initialization and probes. For Ready but slow Pods, inspect effective capacity and downstream waits. Combine faster startup, baseline headroom, and admission control so that work does not accumulate without bounds while scaling catches up.

Budget total database connections before adding Pods

In an original example with one process per Pod and a pool limit of 20 connections, 20 Pods allow up to 400 connections and 30 allow 600. A database capped at 500 is exceeded by the latter before other clients are counted. Include temporary extra processes during rollout or termination, batch jobs, and administrative access in the budget.

Limit	How to count	Decision
Database connections	Sum pool maxima across all simultaneously existing processes	Allocate after reserving capacity for other clients and administration
External API	Calls from all Pods, including live work and retries	Bound aggregate rate and concurrency
Waiting work	Counts and ages in pool waits, application waits, and queues	Bound length and lifetime; define rejection or deferral

Pool maxima describe potential connections, not necessarily connections open continuously. A database can also saturate on locks or I/O below its connection limit. Load-test query latency, pool waits, and timeouts alongside counts, then choose replica limits and pool allocations together.

Validate with comparable traffic and startup conditions

Keep arrival rate, concurrency, payload, and duration comparable before and after a change. Test steady traffic and startup, recording restarts, evictions, memory peaks, throttling, p99 latency, time to Ready, and database waits. Follow business success and completion time beyond a lower CPU utilization number to detect whether the change simply moved the bottleneck.