All posts

Why "save, then publish" breaks

October 25, 2026 · 9 min read

Save the order, then publish an event. A crash between those two lines is enough to leave an order with no confirmation email, or a customer charged twice. This post follows an order through the system to see what to guard, and where.

Wherever it crashes, only one side remains

When an online store takes an order, it saves the order to the database and publishes an "order created" event to a message broker. Stock reservation and the confirmation email run off that event.

The problem is that the database and the broker are separate systems that cannot commit together. A deploy or a node failure can stop the process between any two lines.

Whatever the order, one failure leaves only one side done. A distributed transaction spanning the database and the broker is unavailable or too heavy in most setups.

The outbox: write what you will send in the same transaction

Stop publishing the event directly. Write the business data and "the event to be sent" in the same transaction on the same database. Either both remain or neither does.

BEGIN;
INSERT INTO orders (id, user_id, total, status)
  VALUES ('ord_123', 'u_9', 4800, 'pending');
INSERT INTO outbox (id, aggregate_id, type, payload)
  VALUES ('evt_456', 'ord_123', 'OrderCreated',
          '{"orderId": "ord_123", "total": 4800}');
COMMIT;

A separate process, the relay, reads the outbox table and does the sending. There are two ways to build it.

-- Take unsent rows without competing with other relays
SELECT id, aggregate_id, type, payload
  FROM outbox
 WHERE sent_at IS NULL
 ORDER BY created_at
 LIMIT 100
   FOR UPDATE SKIP LOCKED;

-- Mark them once sent. A crash here means they are sent again after restart
UPDATE outbox SET sent_at = now() WHERE id = ANY($1);

Exactly-once stops at the system boundary

Receivers have the same problem. A consumer processes a message and then commits how far it has read (its offset). If it crashes in between, it processes the same message again after restarting.

Kafka transactions can give exactly-once when reading from Kafka and writing to Kafka. They do not cover side effects outside it, such as calling a payment API or writing to another database.

So accept at-least-once delivery and drop duplicates per side effect. For a side effect in your own database, record the processed message ID in the same transaction.

BEGIN;
INSERT INTO processed_messages (message_id) VALUES ('evt_456')
  ON CONFLICT (message_id) DO NOTHING;
-- Zero rows inserted means already processed: commit without doing anything
UPDATE inventory SET reserved = reserved + 1 WHERE sku = 'sku_1';
COMMIT;
-- Advance the offset after the COMMIT

For outside side effects, pass the order ID as the idempotency key

A payment API lives outside your database, so that method cannot protect it. If the process crashes after charging but before recording the message, a charge with no record remains.

Instead, pass a business ID such as the order ID to the payment API as the idempotency key. If redelivery calls the charge for the same order again, the provider returns the first result.

If your own API accepts idempotency keys

Conversely, if your API is the one being retried, storing the key is not enough. Decide how to respond in these three situations.

Store the result (status code and body) in the same transaction as the business data, and return it for later requests with the same key. Keep keys longer than clients can keep retrying. The Idempotency-Key header draft under discussion at the IETF defines responses for these same situations.

Across services, tie the steps together with a saga

When orders, payments, and inventory each have their own service and database, one transaction cannot cover them. A saga commits each step in its own service, and if a later step fails, it undoes the completed steps with compensating transactions.

The parts built so far become the saga's parts. Send each step's command through the outbox, let receivers drop duplicates by message ID, and make compensation calls idempotent too.

Unlike a database rollback, intermediate states (payment authorized, stock not yet reserved) are visible to other processes. Make "pending" an explicit state so unconfirmed orders never ship.

A checklist for reviews

1. Is anything written to the database and published as separate steps?
2. Does every receiver produce the same result when a message arrives twice?
3. Is the processed-message record written in the same transaction as the side effect?
4. Do writes to external APIs pass a business ID as the idempotency key?
5. Does your own write API define responses for simultaneous, in-progress, and mismatched requests?
6. Does every saga step have a compensation, with irreversible steps after the pivot?

When asked, "How do you keep orders and events consistent?"

1. A dual write leaves only one side done if you crash in between
2. Write the business data and the event in one transaction with an outbox
3. Delivery is at-least-once, so make receivers idempotent by message ID
4. Pass the order ID to the external payment provider as the idempotency key
5. Across services, use a saga, with irreversible steps after the pivot

Saying "we make at-least-once delivery look like once with idempotent processing" shows you understand the boundary far better than "we use exactly-once delivery."