All posts

Is "CPU is only at 80%" really fine?

October 25, 2026 · 9 min read

Sizing a system takes four formulas, all multiplication and division. Using an API that peaks at 3,000 requests per second, this post works out servers, headroom, and connections in turn.

Averages alone do not size a system

Multiplying average QPS by average latency and calling it enough is a common mistake. Outages do not happen at the average. They happen in the peak few minutes, in the slowest 1%, and the moment a zone goes down.

This post estimates, for one API, its concurrency, servers, headroom, and connections, in that order.

Little's law, used three ways

In a stable system, the average number of items inside, L, equals the arrival rate λ times each item's time in the system, W (L = λ × W). It holds whether the inside is a queue or a server, so it applies at every hop.

The second row is the most useful. A pool of 100 with 100 ms responses takes 1,000 requests per second, but if the database slows to 200 ms, capacity halves while traffic stays the same. That explains outages where the pool fills up.

Utilization and waiting follow a curve

Requests arrive unevenly. The higher the utilization, the more often a request arrives while the previous one is still running, and it waits. In the simplest model (M/M/1), with service time S and utilization ρ, the average response time is S ÷ (1 − ρ).

The same 10-point rise has very different effects. From 50% to 60%, 20 ms becomes 25 ms. From 85% to 95%, about 67 ms becomes 200 ms. That is about 27 times the slowdown.

Sizing servers

Consider an API that receives 3,000 requests per second at peak, each using 20 ms of CPU.

1. Work needed: 3,000 × 0.02 s = enough to keep 60 cores busy.
2. Divide by target utilization: with a 60% cap, 60 ÷ 0.6 = 100 cores.
3. Prepare for losing a zone: to keep 100 cores after one of three zones fails, run 50 per zone, 150 in total.

With 150 cores, utilization at the normal peak is only 40%. If you allow 80% only during the outage, the remaining two zones need 75 cores, and 114 in total is enough.

Technology does not decide between 150 and 114. It is an agreement about how slow things may get during an outage.

Watch the slowest 1%, not the average

When a request queries many backends in parallel and waits for all of them, one slow server slows the whole request. If each is slow with probability p and you query n, the share of slow requests is 1 − (1 − p)^n.

Working backward is harsher still. To keep slow requests at 1% while fanning out to 100 servers, each must be slow only about 0.01% of the time. To meet an overall p99, each shard's p99.99 has to meet the same target.

• Fan out less: rethink data placement so each request asks fewer servers.
• Cut off: stop at the deadline, and design the screen to handle missing pieces.
• Hedge: send a second copy to another replica only for requests still pending after p95. The extra load is a few percent (idempotent reads only).

Sizing connections

From the database's side, connections are servers × processes × pool size. Scale the app out and they multiply.

Little's law sizes the pool too. If a server receives 250 requests per second and each holds a connection for 8 ms, it uses 2 on average. Even with a database five times slower, 10 are enough. A pool of 20 only helps once the database is already slow, and more concurrent queries at that point slow it further.

Set the ceiling from the database's side. The HikariCP wiki suggests "database cores × 2 + disks" as a starting point, a little over 30 for 16 cores. Split across 40 app servers, that is less than one each, so when the server count varies widely, put a connection proxy such as PgBouncer or RDS Proxy in between.

One estimation sheet

Using this in an interview

Write the formula first, then plug in numbers, so the interviewer can check your assumptions as you go.

Offering the price of headroom and a way to lower it, as in that last sentence, shows you decide by trade-offs rather than reciting numbers.