All posts

Why a slightly slow database can take everything down

October 25, 2026 · 9 min read

It started with one query that missed an index. Following a payment API from there to a full outage shows the order in which timeout and retry settings come into play.

The setup: from the pay button to the database

Normally the API receives 500 requests per second, and each returns in about 100 ms. One day a query that misses an index ships, and database calls slow down by a few hundred milliseconds.

As triggers go, this is small. Whether everything stops depends on how the settings from here on combine.

Slowing down: every hop's timeout runs on its own

The first thing to grow is the number of requests in flight. It equals arrival rate × time in system (Little's law), so when responses stretch to one second, 50 in flight become 500.

Next, mismatched timeouts create waste. Even after the API gives up at 1 second, an inventory service with a 5-second timeout keeps querying the database for 4 seconds on behalf of nobody. Nothing tells the downstream service that its caller has gone.

Hand out the time budget from the outside in. Then pass the deadline itself downstream instead of fixed values.

// At the API's entry point, set the deadline for the whole request
ctx, cancel := context.WithTimeout(r.Context(), 2500*time.Millisecond)
defer cancel()

// If too little time is left, fail without calling downstream
if deadline, ok := ctx.Deadline(); ok && time.Until(deadline) < 50*time.Millisecond {
	return errDeadlineTooClose
}

// gRPC passes the ctx deadline downstream, so the callee can stop at the same moment
res, err := inventory.Reserve(ctx, req)

Failing: retries multiply

Once timeouts appear, every hop starts retrying. If the client, the API, and the inventory service each make up to 3 attempts, up to 3 × 3 × 3 = 27 attempts reach the database.

The usual 1,000 actions per second become 27,000 attempts per second during the outage, and the database falls over again just as it recovers. There are three fixes.

1. Choose one layer to retry. Every other layer passes failures straight up.
2. Grow the wait exponentially and add random jitter. If everyone retries on the same schedule, the load peaks line up.
3. Cap the total amount of retrying. Add 0.1 token per success, spend 1 per retry, and stop retrying when none are left.

func backoff(attempt int) time.Duration {
	const base, limit = 100 * time.Millisecond, 2 * time.Second
	d := base << attempt // 100ms, 200ms, 400ms, …
	if d > limit {
		d = limit
	}
	// Pick uniformly between 0 and d to spread the retry peaks
	return time.Duration(rand.Int63n(int64(d)))
}

With the third cap, retries never exceed a tenth of successes. In a full outage there are no successes, the tokens run out, and load stops at about 1.1 times normal.

Overflowing: refusing some work early returns more

Suppose a server that handles 1,000 requests per second keeps receiving 1,500. An unbounded queue grows by 500 every second, and the wait grows by half a second every second.

After 4 seconds the wait passes the timeout, and the server works only on requests whose clients have given up. It is busy, yet almost no response succeeds.

Set the queue limit to "tolerable wait × throughput." If a 0.5-second wait is acceptable, that is 1,000 × 0.5 = 500 requests. Anything beyond gets an immediate 429 or 503.

• Drop expired work: check the deadline just before starting, and discard the request if it has passed.
• Prioritize: keep payments flowing, and shed recommendations and view-history logging first.
• Tell the caller: answer "not now" so the sender slows down.

The deadline passed in at the start is exactly what the expiry check needs. Timeout budgets and load shedding share the same deadline.

Not recovering: the trigger is gone, but load stays high

An API that handles 10,000 requests per second normally receives 6,000. For one minute traffic jumps to 15,000, timeouts rise, and clients start trying each request 3 times.

The spike ends and user requests fall back to 6,000, but almost all are tried 3 times, so arriving load stays at 6,000 × 3 = 18,000. It stays above capacity and does not recover on its own.

This is a metastable failure. You cannot prevent every trigger: a spike, a deploy, a lost cache. What design has to break is the mechanism that sustains the load, and here that is retries.

To get out, cut intake at the entrance to 5,000 per second and stop retries. Once responses return in time, let traffic back in about 10% every few minutes. Opening up all at once can send the full load into a cold cache and put you back where you started. Decide this procedure before an outage, and rehearse it.

Is it safe to retry at all? Idempotency keys

Reads can be repeated without harm. Payments cannot. When a request times out, the client cannot tell whether it was never processed or only the response was lost.

If 0.1% of 100,000 daily payments time out after the work is done, about 100 customers a day are charged twice.

The client creates one idempotency key per operation and sends the same key on every retry. The server stores the key and the result in the same transaction as the payment record, and returns the stored result for a repeated key. The key is created by the retrying layer, so keeping retries in one place keeps key handling in one place too.

A checklist for reviews

1. Are timeouts longer toward the outside, and based on measurements such as p99.9?
2. Is the deadline passed downstream, so calls that cannot finish are skipped?
3. Does exactly one layer retry, and have you multiplied out the maximum attempts at the bottom?
4. Do retries use backoff with jitter, with a cap on the total?
5. Is every queue bounded, with expired requests dropped before any work?
6. Do you rehearse cutting intake at the entrance and letting it back in gradually?
7. Do retried writes carry idempotency keys?

When asked, "What if the database you depend on slows down?"

Following this post's order turns the answer into one story.

1. In-flight work grows and the pool fills up (arrival rate × time)
2. Pass the deadline downstream to stop wasted work
3. Retry in one layer, with a cap on the total
4. Refuse work beyond capacity early
5. If you still end up in a bad state, cut intake and let it back in gradually

Adding one number, such as "three layers of 3 attempts means 27 times the load," makes the answer far more convincing.