502s on every deploy

October 25, 2026 · 9 min read

Errors during deploys are easy to ignore because they stop quickly, but they rarely have a single cause. This post follows one pod being replaced and removes the causes one at a time.

The symptom: a few dozen 502s on every deploy

An API runs on 20 pods, replaced one at a time by a rolling deploy. Every deploy throws a few dozen 502s, and the deploy itself takes over 10 minutes.

There is more than one cause. A pod stops, a new pod joins, the schema changes in between, and finally everyone gets the new version. Each stage holds its own source of errors.

Replacing one pod. This post follows these stages in order

Old pod stops → New pod joins → Schema changes → Release to everyone

Stopping: do not exit before leaving the rotation

When Kubernetes deletes a pod, two things start at the same time: stopping the pod (preStop, then SIGTERM) and updating routing (removing the pod from the endpoints, which every node and Ingress then picks up). Neither waits for the other.

Routing takes a few seconds to catch up. A pod that exits as soon as it gets SIGTERM keeps receiving requests for those seconds, and they fail with connection errors.

The right shutdown order when a pod is deleted

Kubernetes: Starts deleting the pod
Load balancer: Removes it from endpoints (takes seconds)
Pod: Waits 5 s in preStop, serving what arrives
Kubernetes: Sends SIGTERM
Pod: Stops accepting, finishes in-flight requests
Pod: Closes database connections and exits

Keep the total under the grace period before a forced kill (terminationGracePeriodSeconds, 30 seconds by default). The grace period includes preStop, so 5 seconds in preStop plus up to 20 for in-flight requests fits.

YAML: wait in preStop before SIGTERM arrives

spec:
  terminationGracePeriodSeconds: 30   # includes the preStop time
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            # Wait until removal from the endpoints has taken effect
            command: ["sleep", "5"]

Go: on SIGTERM, stop accepting and wait for in-flight requests

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()
go srv.ListenAndServe()

<-ctx.Done() // SIGTERM arrives after preStop has finished
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx) // stop accepting, wait up to 20 s for in-flight requests
db.Close()

Not stopping: SIGTERM never reaches the app

The shutdown code is in place, yet each pod stops after almost exactly 30 seconds. That "exactly the grace period" is the clue. The app never received SIGTERM and was SIGKILLed when time ran out.

The cause is an ENTRYPOINT in shell form. PID 1 becomes /bin/sh, and the shell does not pass SIGTERM on to the app. The shutdown code had never run.

Dockerfile: shell form and exec form

# Shell form: PID 1 is /bin/sh, and SIGTERM never reaches the app
ENTRYPOINT node server.js

# Exec form: the app is PID 1 and receives SIGTERM directly
ENTRYPOINT ["node", "server.js"]

Time to replace 20 pods one at a time

Shell form (30 s wait, then SIGKILL): over 10 min
Exec form (stops in about 1 s): about 2 min

PID 1 ignores signals it has no handler for, so the app needs its own SIGTERM handler. If the app spawns child processes, make a tiny init such as tini PID 1.

Joining: get health-check depth right

A new pod stays out of the rotation until it is ready; that is the readiness probe's job. With maxUnavailable: 0 and maxSurge: 1, old pods stop only after new ones are ready, so capacity never drops during the rollout.

The danger is a check that goes too deep. Suppose all 20 liveness probes query the database and time out after 1 second. When a nightly batch slows the database to 2 seconds, every pod restarts at once, they all reconnect to the database together, and the whole service goes down.

Probe	On failure	May check	Must not check
Liveness	Restart	Inside the process (a deadlock, say)	External dependencies such as the database
Readiness	Remove from rotation	This pod's own readiness	Dependencies every pod shares

Watch dependencies shared by every pod with monitoring and alerts, not with decisions to remove or restart.

Stopping together: node replacement

Pods also stop outside deploys, when a cluster upgrade drains a node. If two of three pods share a node, both are evicted at once, and every request lands on the remaining one.

A PodDisruptionBudget (PDB) sets how many may stop at once. Set it to zero, though, and evictions never proceed; some managed services then force them after a timeout. For a single-replica service, add replicas rather than blocking with a PDB.

YAML: at most one pod down at a time

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

The PDB decides how many pods stop; the shutdown sequence decides how one pod stops. You need both for node replacement to be error-free.

Old and new run together: split schema changes

For the few minutes a rolling deploy takes, old and new apps share the database. Rename name to full_name now, and old pods still running cannot read name and return 500s. No fix to shutdown removes these errors.

Split the change into steps that break neither version (expand-contract).

Release	Database	App
1	Add the full_name column	No change
2	—	Write both name and full_name
3	Backfill existing rows in small batches	Write both
4	—	Read full_name
5	—	Read and write only full_name
6	Drop the name column	—

Check that the previous app version still runs safely in the middle of every release. Set lock_timeout on the ALTER that adds the column, too.

Releasing: check with a few users before everyone

By now, correct code swaps in without errors. What remains is code that is itself wrong. A checkout change went to every pod, the error rate jumped to 5%, and everyone was affected for the 15 minutes until rollback.

An incident's impact is roughly share of users affected × time until rollback.

Impact of an incident (share × time, log scale)

Release to every pod: 100% × 15 min
1% canary: 1% × 15 min
1% canary with automatic stop: 1% × 5 min

Define stages: widen from 1% to 10%, 50%, then 100%, waiting at each.
Compare with the old version: an overall average buries a 1% canary's regression.
Separate deploy from release: ship behind a feature flag, and turn the flag off if something breaks.

A checklist for reviews

After a few seconds in preStop, does the pod stop accepting, finish in-flight requests, then close connections?
Is ENTRYPOINT in exec form, and have you measured the time from SIGTERM to exit?
Does the liveness probe look only inside the process?
Do settings keep capacity from dropping during the rollout?
Do multi-replica services have a PDB, with pods spread out?
Are schema changes split into steps that work with both old and new apps?
Does each new version go to a few users first, with an automatic stop on its metrics?

When asked, "How do you deploy without downtime?"

Do not stop at "rolling deploys." Walk through one pod being replaced.

Stopping: wait to leave the rotation, then stop accepting and finish in-flight requests
Joining: enter the rotation only when ready, and keep shared dependencies out of health checks
While old and new run together: split schema changes into add, migrate, remove
For code that is wrong: release to a canary and stop automatically on its metrics

Estimating impact as "share × time" and saying you can cut it to a hundredth explains why the effort is worth it.