Clear a message queue backlog: capacity, retries, and safe DLQ recovery

October 25, 2026 · 10 min read

More workers may leave a queue growing, while a DLQ redrive can trigger another outage. Use Amazon SQS examples to calculate drain time, measure business completion, preserve idempotency, and replay within downstream capacity.

What to inspect when a queue will not shrink

A queue buffers temporary excess traffic for later processing. Fast acknowledgement does not establish that an order or notification will finish on time. Define an acceptance-to-completion deadline, then align new arrivals, distinct completions, unfinished jobs, and the age of the oldest unfinished business operation on one timeline.

Observation	Possible state	Next check
Arrivals exceed completions and age increases	Insufficient capacity or a slow dependency	Active workers, database waits, API limits
Attempts increase but completions stay flat	Retries consume processing slots	Success by attempt number and error category
Source queue shrinks while DLQ grows	Failures are being quarantined	Unfinished business operations and recovery owner
Only a subset remains old	Invalid input, hot keys, or ordering constraints	Failure samples, key skew, and dependencies

Amazon SQS queue counts and ages are approximate. The oldest-message metric in standard queues can exclude repeatedly failing messages; moving to a DLQ changes the age reference. Preserve the original acceptance timestamp in the payload or business ledger so that transfers do not erase the history.

Calculate drain time: B / (μ − λ)

The following is an original hypothetical calculation. Assume a stable job mix and processing-time distribution, constant arrivals λ and effective completion capacity μ, and no departures through quarantine or expiry. If μ exceeds λ, draining B unfinished jobs takes B / (μ - λ). Merely reducing arrivals to match capacity leaves no spare capacity to clear existing work.

Accepting 1,200 jobs/second while completing 1,000/second for 15 minutes leaves 180,000 jobs. If arrivals then fall to 800/second, the backlog shrinks by 200/second and drains in 15 minutes. Keeping arrivals at 1,200/second but raising completion capacity to 1,800/second drains it in five minutes.

Python 3: drain time under constant rates

# Python 3: illustrative constant-rate model, not a latency prediction
backlog = (1200 - 1000) * (15 * 60)
print(backlog)  # 180000 logical jobs

def drain_seconds(backlog, arrival_per_second, completion_per_second):
    if completion_per_second <= arrival_per_second:
        return None  # no finite drain time at these constant rates
    return backlog / (completion_per_second - arrival_per_second)

print(drain_seconds(backlog, 800, 1000))   # 900.0 seconds
print(drain_seconds(backlog, 1200, 1800))  # 300.0 seconds
print(drain_seconds(backlog, 1200, 1000))  # None

Calculated drain time for the same 180,000 jobs, in seconds, with constant arrival and completion rates.

800 arrivals/s; 1,000 completions/s: 900 seconds (15 minutes)
1,200 arrivals/s; 1,800 completions/s: 300 seconds (5 minutes)

Separate attempt throughput from business completions

Executing 2,000 attempts/second does not add completion capacity if they repeat the same failures. Record initial acceptances, all attempts, first successes by business ID, and quarantines separately. Inspect both queued and in-flight work; receiving a message is not completion.

Queue departures can include DLQ transfers and expiry, so they cannot directly supply μ in the previous model. Keep pending, successful, and investigation-required states in a business ledger. Decide whether an expired notification may be discarded or an order requires compensation. Retention expiry alone cannot establish success.

A longer visibility timeout does not prevent duplicate effects

After an SQS receive, the visibility timeout temporarily hides a message. Work running beyond that deadline can be delivered again without establishing that the original worker has stopped. Extend visibility when appropriate, while protecting side effects against repeated delivery.

For effects within one database, commit a unique tenant/operation ID, the state transition, and the completion record in one transaction. Establish what rolls back together.
For external payments, check the provider's idempotency key and retention rules. Resolve ambiguous timeouts through lookup and reconciliation using the same operation ID, rather than executing under a new ID.
Preserve the business operation ID across DLQ transfers and replay. SQS redrive assigns a new messageID, so delivery IDs alone are insufficient for deduplication.
Retain completion records for the recovery replay horizon as well as ordinary retries, considering how long external outcomes can be queried.

Check shared downstream limits before adding workers

Suppose a downstream API can safely accept 1,000 calls/second in total, live work uses 700, and retries use 100. Replay has at most 200 calls/second available. That is a calculated ceiling assuming one call per job and no other users; leave additional operating headroom. At two calls per job, those 200 calls support at most 100 replayed jobs/second.

More instances can increase database connections and outbound calls. A 200-calls/second limiter on each of ten processes does not enforce a global 200-calls/second limit. Use shared admission control or allocated budgets, and bound waiting work and its lifetime. Limit retry counts, elapsed retry budgets, and spacing when a dependency is saturated.

A procedure for safe DLQ recovery

Inspect business IDs, original timestamps, attempts, and error categories. Identify whether credentials, malformed input, or temporary overload needs correction.
Apply the fix and verify duplicate handling and ambiguous external outcomes. Provide a separate correction or quarantine path for invalid data.
Estimate spare capacity including live work and retries, and start at a low replay rate. SQS redrive does not filter or modify messages during movement; perform required corrections separately.
Increase the rate only after checking successful business outcomes, database waits, API limiting, and live-work latency. Stop replay on degradation and track work already moved.
Reconcile completed, compensated, and investigation-required operations before closing recovery. Record an owner and deadline for unresolved work, beyond the falling DLQ count.

Replay while live traffic continues in recovery tests

Slow a dependency temporarily, create a backlog, and restore service. A test with live arrivals stopped misses replay competing with normal work. Include old message schemas, repeated business operations, a worker crash immediately after processing, and ambiguous external API results. Check both duplicate effects and abandoned operations.

Define acceptance using live-work deadlines, drain time, downstream waits, zero duplicate effects, and visible unresolved cases alongside queue depth. Updating the model with measured completion capacity makes worker counts, admission limits, and replay rates actionable during the next incident.