The service commits to its DB then crashes before publishing the event — DB and broker are out of sync. Diagnose and fix.
An order service writes the order to its database, then publishes an OrderCreated event to the broker as a second, separate step. Under load, some orders exist in the database but the matching event never reached the broker, and downstream services are missing them. The log below shows one such request.
Constraint: the database and the broker are independent systems — you cannot assume a distributed transaction across them.
12:04:01 BEGIN TX
12:04:01 INSERT orders(id=8821, status=NEW) -> ok
12:04:01 COMMIT TX -> ok
12:04:01 publish OrderCreated(id=8821) ...
12:04:02 process killed (OOM) <-- crash before broker ACK
--- broker topic orders.events: no record for id=8821 ---Diagnose why the DB and the broker diverge, and describe the fix.
The DB commit and broker publish are separate writes, no shared transaction; a crash between them saves the order but loses the event (dual write). Fix with a transactional outbox: write it to a table in the same transaction; a relay publishes it.
- ✗Thinking a publish-retry on restart fixes it, though the service has no record the event is unsent
- ✗Publishing before the DB commit, which emits events for orders that may roll back
- ✗Reaching for a distributed two-phase commit instead of a local-transaction outbox
- →How does a relay reading the outbox avoid publishing an event twice?
- →Why is reading the DB change log often preferred over polling the outbox table?
Cause
This is the classic dual-write problem. The DB commit and the broker publish are two independent operations with no shared transaction. The process died (OOM) between COMMIT and publish, so the order exists in the DB but no event does. Retrying the publish on restart does not help — after the crash the service has no memory that the event was never sent.
Fix — transactional outbox
Write the event into an outbox table in the same local transaction as the order, so the order and the event record commit atomically — both or neither. A separate relay reads outbox and publishes to the broker, marking each row as sent (at-least-once plus an idempotent consumer on the reader side).
BEGIN;
INSERT INTO orders (id, status) VALUES (8821, 'NEW');
INSERT INTO outbox (id, topic, payload, published)
VALUES (gen_random_uuid(), 'orders.events', '{"orderId":8821}', false);
COMMIT;
-- relay: SELECT ... FROM outbox WHERE published = false -> publish -> UPDATE published = trueNow the order write and the intent to publish are indivisible; the relay guarantees delivery even after a crash.
