SQS vs Pub/Sub vs Azure Service Bus: delivery guarantees, ordering, and replay
Compare AWS SQS and SNS, Google Cloud Pub/Sub, and Azure Service Bus for work distribution, event fanout, deduplication, ordering, backlog, and dead-letter recovery. Separate exactly-once delivery from preventing duplicate business effects.
SQS, Pub/Sub, and Service Bus: work distribution versus event fanout
A confirmed order must reach shipping and analytics. Ten shipping workers should share each shipping job rather than all perform it. Analytics, however, needs its own copy of the event. Separate competing-consumer work distribution from fanout to independent business consumers.
| Purpose | AWS | Google Cloud | Azure |
|---|---|---|---|
| Share work among shipping workers | Multiple consumers receive from one SQS queue | Multiple subscribers receive from one subscription | Multiple receivers consume one Service Bus queue |
| Deliver separately to shipping and analytics | SNS topic fans out to a separate SQS queue per business consumer | Pub/Sub topic with a separate subscription per business consumer | Service Bus topic with a separate subscription per business consumer |
Mixing shipping and analytics workers in one competing-consumer queue or subscription does not give both every event. Isolate backlog and retries by business purpose so slow analytics does not stop shipping. Additional subscriptions also add delivery, retention, and consumer costs.
What do deduplication and exactly-once actually make singular?
| Feature or mode | Documented boundary | Design separately |
|---|---|---|
| SQS Standard | At-least-once delivery; duplicates and reordering can occur | Avoid repeating the business operation on redelivery |
| SQS FIFO send deduplication | Suppresses duplicate sends within a 5-minute window using deduplication IDs or the configured equivalent | Sends after the window and reprocessing after a crash before deletion |
| Pub/Sub exactly-once delivery | Enabled pull subscriptions, within a region; no redelivery after a successfully confirmed acknowledgment | Not supported for push or export; valid redelivery after ack expiry and external effects |
| Service Bus duplicate detection | Enabled Standard or Premium entities deduplicate send IDs within a configured window | Receiver reprocessing and external effects; check PartitionKey when partitioned |
Service Bus allows a duplicate-detection window from 20 seconds to 7 days, with a 10-minute default. Pub/Sub's delivery guarantee also does not identify two separately published messages as the same business event. In every case, a consumer that stops immediately after a shipping API succeeds leaves an outcome the messaging service alone cannot resolve.
- Messaging service: Deliver ship-123
- Carrier API: Shipping request succeeds
- Worker: Crash before saving the result and settling delivery
- Messaging service: Redeliver after expiry or another valid trigger
- Carrier API: Reconcile and deduplicate with the same ship-123
Bridge order storage and publication with an outbox; deduplicate by operation
A crash after saving an order but before publishing leaves shipping uninformed. Consider writing the order and its outbox event in one database transaction, then publishing through a relay or CDC. Manage unsent-event retention, relay restarts, retries, and oldest-unsent age. Publication can repeat if recording a successful send fails, so consumers still need deduplication.
- For an effect entirely inside your database, include a unique processed-event record and the business update in the same transaction, so concurrent deliveries cannot both update it.
- For the external carrier API, use its guarantees, such as the original shipping operation ID as an idempotency key. Reconcile unknown outcomes with that ID; a local processed flag alone does not prove success.
- Keep the business operation ID separately from the delivery message ID. Align retention with normal retries, dead-letter replay, and the external API's idempotency-key lifetime.
Decide whether ordering belongs to an order or a customer
| Candidate | Ordering unit and conditions | Concurrency implication |
|---|---|---|
| SQS FIFO | Ordering within MessageGroupId | Parallelize different groups and preserve processing order within each group in consumers |
| Pub/Sub ordering | Same ordering key on an enabled subscription, published in the same region | No global order across keys; measure slow keys separately |
| Service Bus sessions | Same SessionId in Standard or Premium | Receive by session and handle lock renewal and expiry |
Putting everything under one key lets a slow order block unrelated work. If creation, amendment, and cancellation must remain ordered per order, use order IDs to retain parallelism. Arrival order need not equal valid business order, so also verify event versions and current state. Define ordering behavior separately when isolating work in a DLQ and returning it later.
Include retries, subscribers, and downstream limits in backlog and cost
Assume 900 arrivals per second and successful processing capacity of 700 per second for 10 minutes. Backlog grows by (900 − 700) × 600 = 120,000 messages. If arrivals then fall to 500 per second, spare capacity is 200 per second; under constant rates, draining takes 120,000 ÷ 200 = 600 seconds, or 10 minutes. More workers cannot raise successful throughput above a carrier API limit of 700 per second.
- Monitor oldest-unprocessed age, business completion time, and failures as well as message count. Check whether recovery can finish before retention expires.
- Map retries, receives, settlement or deletion, empty polling, delivered bytes, subscriptions, and retention to actual billing rules. A publish price alone does not establish total cost.
- Include queue and plan quotas, ordering-key skew, consumer compute, private connectivity, and cross-region transfer.
A DLQ contains unfinished work: make safe replay an adoption requirement
Sending a failed message to a DLQ does not complete shipping. If the carrier accepted a request but saving the result failed, fixing the cause and replaying everything can create duplicate shipments. SQS redrive assigns a new messageID, so reconcile with the original shipping operation ID. Check how each provider handles identifiers and expiry during replay.
- Assign ownership, notification routing, retention, and operation IDs for investigation. An empty normal queue can still leave unfinished work in the DLQ.
- Reconcile the cause and existing effects. Repair records for completed operations; review expired or order-sensitive work separately from normal replay.
- Replay only necessary work, starting with a small sample at a low rate. Increase while observing normal latency and downstream failures, with explicit stop conditions.
- Inject crashes after effects, before settlement, and on lock expiry. Verify duplicate effects and missing work against business results, then compare operations and cost for configurations that pass.
Primary sources for delivery, ordering, and deduplication boundaries
- AWS — SQS standard queues
- AWS — SNS fanout to SQS
- AWS — SQS FIFO delivery logic
- AWS — SQS FIFO deduplication
- Google Cloud — Pub/Sub service overview
- Google Cloud — Pub/Sub exactly-once delivery
- Google Cloud — Pub/Sub message ordering
- Microsoft Learn — Service Bus overview
- Microsoft Learn — Service Bus duplicate detection
- Microsoft Learn — Service Bus sessions
- AWS — Transactional outbox pattern
- AWS — SQS dead-letter redrive
Tips this post digs into
- Separate DB updates and message sends can succeed independently
- Effects outside an exactly-once guarantee need their own deduplication
- Retries of non-idempotent writes need deduplication such as idempotency keys
- A queue does not remove overload; it defers it, and the debt comes due
- Moving a failed message aside does not complete the business operation