Messaging & Delivery Guarantees
Broker reliability and delivery semantics — Kafka vs RabbitMQ, partitions/offsets/consumer groups, delivery guarantees, idempotent consumers, dead-letter queues, the outbox and saga patterns, and ordering.
15 questions
JuniorTheoryVery commonWhat are a producer and a consumer, and what does the broker do between them?
What are a producer and a consumer, and what does the broker do between them?
A producer publishes messages; a consumer reads and processes them. The broker sits between them — accepting, storing durably, and delivering messages. This decouples the two sides: neither calls the other directly, and they need not be online at once.
Common mistakes
- ✗Thinking the broker only routes addresses and does not store messages itself
- ✗Assuming producer and consumer must both be online simultaneously to exchange a message
- ✗Confusing which side publishes and which side subscribes
Follow-up questions
- →What does the broker give you that a direct producer-to-consumer socket cannot?
- →How does durable storage in the broker help when a consumer is temporarily down?
JuniorTheoryVery commonWhat is a queue versus a topic, and how does a consumer read from each?
What is a queue versus a topic, and how does a consumer read from each?
A queue gives each message to exactly one consumer — competing consumers share the load and a message leaves once handled. A topic broadcasts each message to every subscriber, so many consumers read the same message, each at its own read position.
Common mistakes
- ✗Assuming a topic, like a queue, removes a message once one consumer has read it
- ✗Believing extra consumers on a queue multiply delivery instead of sharing the load
- ✗Thinking every subscriber on a topic shares one read position rather than tracking its own
Follow-up questions
- →When would you pick competing consumers on a queue over a broadcast topic?
- →How does a consumer on a topic resume after a restart without losing its place?
MiddleTheoryVery commonWhich delivery guarantees exist, and which does the streaming platform Kafka actually give you?
Which delivery guarantees exist, and which does the streaming platform Kafka actually give you?
Three: at-most-once (loses messages on failure), at-least-once (may duplicate on retry), and exactly-once (neither). Kafka offers true exactly-once only inside its own transactions; the moment you touch an external system you fall back to at-least-once plus idempotency.
Common mistakes
- ✗Believing exactly-once is automatic rather than bounded to the broker's own transactions
- ✗Swapping which guarantee loses messages and which one duplicates them
- ✗Assuming exactly-once extends to external side effects, so idempotency is unnecessary
Follow-up questions
- →Why does an external side effect force you back to at-least-once plus idempotency?
- →How does at-least-once combined with a dedup key approximate exactly-once in practice?
MiddleDesignCommonYour consumers are running four hours behind the producer — lag keeps growing and downstream data is stale. Design your investigation and your fix. Describe what you would measure to locate the bottleneck (consumer throughput versus incoming rate, per-partition lag, processing time per message, rebalances, errors), which levers you would pull to close the gap (partition count and consumer instances, batch size, slow downstream calls, poison messages parked to a DLQ), and how you tell a temporary spike apart from a consumer simply under-provisioned for steady load.
Your consumers are running four hours behind the producer — lag keeps growing and downstream data is stale. Design your investigation and your fix. Describe what you would measure to locate the bottleneck (consumer throughput versus incoming rate, per-partition lag, processing time per message, rebalances, errors), which levers you would pull to close the gap (partition count and consumer instances, batch size, slow downstream calls, poison messages parked to a DLQ), and how you tell a temporary spike apart from a consumer simply under-provisioned for steady load.
Measure if throughput trails the rate and where time goes — per-partition lag, per-message time. Scale consumers to the partition count, raise it if capped, batch I/O, park poison messages. Steady growth is under-provisioning; spikes self-drain.
Common mistakes
- ✗Adding consumers beyond the partition count, expecting throughput past the parallelism ceiling
- ✗Skipping the backlog by resetting offsets to latest, silently dropping unprocessed messages
- ✗Blaming the producer's rate without measuring per-message processing time or downstream latency
Follow-up questions
- →Why does adding consumers past the partition count stop helping throughput?
- →How do you tell a transient traffic spike from sustained under-provisioning?
MiddleDesignCommonA consumer processes order events. One message fails five times in a row — every retry throws the same error. Design the retry-and-DLQ policy for this consumer. Specify how many times to retry and with what backoff, when a message moves to the dead-letter queue (a sidelined queue for un-processable messages), what metadata travels with it, and who watches the DLQ and acts on its contents. State how you avoid blocking the partition while one poison message is retried, and how an operator later reprocesses a fixed message.
A consumer processes order events. One message fails five times in a row — every retry throws the same error. Design the retry-and-DLQ policy for this consumer. Specify how many times to retry and with what backoff, when a message moves to the dead-letter queue (a sidelined queue for un-processable messages), what metadata travels with it, and who watches the DLQ and acts on its contents. State how you avoid blocking the partition while one poison message is retried, and how an operator later reprocesses a fixed message.
Retry with exponential backoff and jitter for transient faults; after a bounded count, move it to a dead-letter queue, not blocking the partition. Attach payload, error, and attempt count. An owning team is alerted, fixes the cause, and replays.
Common mistakes
- ✗Retrying a poison message forever in place, blocking the partition behind it
- ✗Treating the DLQ as fire-and-forget with no owner, alert, or replay path
- ✗Discarding the payload and error metadata, leaving failures impossible to diagnose
Follow-up questions
- →How do you distinguish a transient fault worth retrying from a poison message?
- →What guards a naive replay from re-triggering the same failure in a loop?
MiddleTheoryCommonWhat is an idempotent consumer, and how do you implement deduplication of redelivered messages?
What is an idempotent consumer, and how do you implement deduplication of redelivered messages?
An idempotent consumer reaches the same end state however often a message arrives, so at-least-once redelivery is safe. Deduplicate: give each a stable id, record processed ids, skip any already seen — ideally in the side effect's transaction.
Common mistakes
- ✗Thinking the broker's exactly-once flag removes the need for consumer-side dedup
- ✗Deduplicating by arrival time rather than by a stable message or business id
- ✗Recording the processed id separately from the side effect, so a mid-way crash still duplicates
Follow-up questions
- →Why should the dedup record and the side effect share one transaction?
- →How do you keep the processed-id store from growing without bound?
MiddleTheoryCommonHow do the message brokers Kafka and RabbitMQ differ in model, retention, routing, and ordering?
How do the message brokers Kafka and RabbitMQ differ in model, retention, routing, and ordering?
Kafka is a partitioned log: consumers pull, messages persist after reading, routing is by partition key, and order holds per partition. RabbitMQ pushes to competing consumers, deletes on ack, routes flexibly via exchanges, and keeps order only within a single-consumer queue.
Common mistakes
- ✗Thinking Kafka deletes a message once a consumer reads it, like a classic queue
- ✗Believing either broker guarantees total ordering across all partitions or consumers
- ✗Assuming RabbitMQ routes by partition key rather than through exchanges and bindings
Follow-up questions
- →Which workload pushes you toward Kafka's retained log over RabbitMQ's delete-on-ack?
- →How does RabbitMQ's exchange-and-binding routing add flexibility Kafka partitions lack?
MiddleTheoryCommonWhat is a saga, and how do an orchestration saga and a choreography saga differ?
What is a saga, and how do an orchestration saga and a choreography saga differ?
A saga keeps data consistent across services without a distributed transaction — a chain of local transactions, each with a compensation to undo it if a later step fails. Orchestration uses a central coordinator to drive each step; choreography has none, so each service reacts to the previous event.
Common mistakes
- ✗Equating a saga with a distributed two-phase commit rather than local transactions plus compensation
- ✗Swapping orchestration (central coordinator) and choreography (event-driven, no coordinator)
- ✗Assuming a failed step rolls back via the database instead of an explicit compensating action
Follow-up questions
- →When does a central orchestrator's visibility outweigh choreography's looser coupling?
- →Why must a compensating action be designed for effects that cannot simply be rolled back?
MiddleTheoryCommonIn the streaming platform Kafka, what are a partition, an offset, and a consumer group, and how do they interact?
In the streaming platform Kafka, what are a partition, an offset, and a consumer group, and how do they interact?
A partition is an ordered, append-only slice of a topic, and parallelism equals the partition count. An offset is a message's position in its partition, committed per group. A consumer group splits partitions so each is read by exactly one member.
Common mistakes
- ✗Treating the offset as a global counter rather than per-partition and per-group
- ✗Thinking all members of a consumer group receive every message, like a broadcast
- ✗Believing partitions are replicas for durability rather than the unit of parallelism
Follow-up questions
- →Why does the partition count cap the parallelism of a single consumer group?
- →What happens to partition assignment when a group member joins or crashes?
MiddleTheoryCommonWhat is the transactional outbox pattern, and which dual-write problem does it solve?
What is the transactional outbox pattern, and which dual-write problem does it solve?
Dual write: a service must update its database and publish an event, but two systems cannot commit atomically, so a crash between them causes inconsistency. Outbox writes the event to a table in the same local transaction as the state change; a relay later reads that table and publishes it.
Common mistakes
- ✗Thinking outbox needs a distributed two-phase commit rather than one local transaction
- ✗Publishing the event before the database commit, which can emit events for rolled-back state
- ✗Placing the outbox inside the broker rather than in the service's own database
Follow-up questions
- →How does a relay reading the outbox table avoid missing or duplicating events?
- →Why is polling the outbox table often replaced by reading the database change log?
MiddleTheoryOccasionalA consumer just read two messages in the streaming platform Kafka. How does that same consumer read them again?
A consumer just read two messages in the streaming platform Kafka. How does that same consumer read them again?
Kafka retains messages and tracks an offset instead of deleting on read, so the consumer moves back: it calls seek to an earlier offset (or timestamp) and polls again. Nothing changes server-side; re-reading is a client-side offset move.
Common mistakes
- ✗Thinking Kafka deletes a message on read, so re-reading needs the producer to republish
- ✗Believing offsets can only advance and can never be moved backward by the client
- ✗Assuming re-delivery is a broker request rather than a client-side
seek
Follow-up questions
- →How would you re-read only messages after a specific timestamp rather than from the start?
- →What risk does committing the offset before processing create for re-reads?
SeniorDesignOccasionalDesign compensating transactions for a booking saga with three steps: reserve seat, charge card, issue ticket. Step three, issue ticket, fails. Describe the compensating action for each already-completed step and the order they run, what state each service must keep to compensate correctly, why a compensation is not the same as a database rollback, and how the design stays correct if a compensating action itself fails or a step is retried.
Design compensating transactions for a booking saga with three steps: reserve seat, charge card, issue ticket. Step three, issue ticket, fails. Describe the compensating action for each already-completed step and the order they run, what state each service must keep to compensate correctly, why a compensation is not the same as a database rollback, and how the design stays correct if a compensating action itself fails or a step is retried.
Compensate completed steps in reverse: refund the card, then release the seat. Each compensation is a forward transaction that undoes its step — not a DB rollback, locals already committed. Compensations keep ids and must be idempotent and retryable.
Common mistakes
- ✗Expecting a database rollback to undo already-committed steps instead of explicit compensations
- ✗Running compensations in forward order rather than reverse of the completed steps
- ✗Assuming compensations run once, so skipping idempotency and retry-safety
Follow-up questions
- →Why must a compensation be idempotent if the saga coordinator retries it?
- →How do you handle a compensation that itself fails partway, like a refund that errors?
SeniorDesignOccasionalA payment event is delivered twice and the customer is charged twice. Walk every layer where this double charge could have been prevented — from the broker's delivery semantics, through the consumer, down to the payment side effect and the external payment provider. For each layer say what mechanism stops the duplicate, or why that layer cannot, and conclude with the single layer you would rely on as the ultimate safeguard and why.
A payment event is delivered twice and the customer is charged twice. Walk every layer where this double charge could have been prevented — from the broker's delivery semantics, through the consumer, down to the payment side effect and the external payment provider. For each layer say what mechanism stops the duplicate, or why that layer cannot, and conclude with the single layer you would rely on as the ultimate safeguard and why.
At-least-once makes duplicates expected; no broker setting removes them. The broker cuts redelivery, the consumer dedups on a stable id, the charge enforces an idempotency key. Rely finally on idempotency at the charge — layers above can duplicate.
Common mistakes
- ✗Believing the broker's exactly-once mode covers the external payment call
- ✗Deduplicating by arrival time rather than a stable event id and idempotency key
- ✗Trusting the ack to prevent redelivery instead of idempotency at the money side effect
Follow-up questions
- →Why can't the broker's acknowledgement alone guarantee the charge happens once?
- →Where should the idempotency key originate so a retry reuses the same one?
SeniorDebuggingOccasionalThe service commits to its DB then crashes before publishing the event — DB and broker are out of sync. Diagnose and fix.
The service commits to its DB then crashes before publishing the event — DB and broker are out of sync. Diagnose and fix.
The DB commit and broker publish are separate writes, no shared transaction; a crash between them saves the order but loses the event (dual write). Fix with a transactional outbox: write it to a table in the same transaction; a relay publishes it.
Open full question →Common mistakes
- ✗Thinking a publish-retry on restart fixes it, though the service has no record the event is unsent
- ✗Publishing before the DB commit, which emits events for orders that may roll back
- ✗Reaching for a distributed two-phase commit instead of a local-transaction outbox
Follow-up questions
- →How does a relay reading the outbox avoid publishing an event twice?
- →Why is reading the DB change log often preferred over polling the outbox table?
SeniorDesignOccasionalKafka guarantees ordering only within a single partition, but your service must process every event for a given customer in the exact order it occurred, across a 12-partition topic under heavy load. Design how you preserve per-customer ordering while still using all 12 partitions for throughput. Explain how events reach the right partition, what happens to global ordering across different customers, how a partition-count change would affect existing keys, and one failure mode your design must tolerate — for example a single hot customer.
Kafka guarantees ordering only within a single partition, but your service must process every event for a given customer in the exact order it occurred, across a 12-partition topic under heavy load. Design how you preserve per-customer ordering while still using all 12 partitions for throughput. Explain how events reach the right partition, what happens to global ordering across different customers, how a partition-count change would affect existing keys, and one failure mode your design must tolerate — for example a single hot customer.
Use the customer id as the partition key: a customer's events hash to one ordered partition, while others spread across all 12 for throughput. Cross-customer order is lost; per-key order holds. Repartitioning remaps keys; a hot key overloads one.
Common mistakes
- ✗Believing Kafka preserves order across partitions when messages share a key
- ✗Expecting global ordering across customers rather than only per-key ordering
- ✗Ignoring that changing the partition count remaps keys and breaks per-key continuity
Follow-up questions
- →How would you relieve a single hot customer that saturates its one partition?
- →Why does increasing the partition count later threaten existing per-key ordering?
