Messaging
Message brokers, Kafka, delivery guarantees, the outbox pattern, dead-letter queues, and idempotency in an event-driven system.
12 questions
SeniorDesignVery commonYou are building a payment-charge API in Go that calls an external payment provider and records the result in your own database. Clients on flaky networks retry a charge they are unsure about, and a load balancer may deliver two copies of one request to two instances at once. Requirements:
- A customer is charged exactly once for a given logical payment attempt, however many times the request is retried or duplicated.
- Two concurrent duplicate requests for the same attempt must not both reach the provider — the race must resolve to a single charge.
- A retry after a successful charge must return the original result, not a fresh charge or an error.
- If the process crashes after the provider charged the card but before the local DB write commits, a later retry must not double-charge.
Specify what the client sends to identify a retry, where you store charge state, and the ordering between the provider call and your own writes.
You are building a payment-charge API in Go that calls an external payment provider and records the result in your own database. Clients on flaky networks retry a charge they are unsure about, and a load balancer may deliver two copies of one request to two instances at once. Requirements: - A customer is charged exactly once for a given logical payment attempt, however many times the request is retried or duplicated. - Two concurrent duplicate requests for the same attempt must not both reach the provider — the race must resolve to a single charge. - A retry after a successful charge must return the original result, not a fresh charge or an error. - If the process crashes after the provider charged the card but before the local DB write commits, a later retry must not double-charge. Specify what the client sends to identify a retry, where you store charge state, and the ordering between the provider call and your own writes.
Require the client to send an Idempotency-Key. On charge, INSERT a row keyed by it inside a transaction with a unique constraint; the first request wins and performs the charge, while a duplicate hits the constraint and returns the stored result instead of charging again. Persist the response with the key so retries are deterministic, and only call the payment provider once the row is reserved.
Common mistakes
- ✗Using a
SELECT-then-INSERTcheck instead of a unique constraint, which races and lets two retries both charge - ✗Calling the payment provider before the key row is reserved, so a crash between the call and the insert double-charges
- ✗Deduplicating on a request-body hash with a TTL instead of a client key, which expires and re-charges a slow retry
Follow-up questions
- →Why does a unique constraint beat a
SELECT-then-INSERTcheck under concurrent duplicates? - →If the process crashes after charging the provider but before committing, how do you avoid a double charge on retry?
JuniorTheoryCommonWhat is event-driven architecture and what problem does it solve?
What is event-driven architecture and what problem does it solve?
Components communicate by emitting and consuming events through a broker instead of calling each other directly. This gives loose coupling and asynchrony — a producer does not know its consumers — at the cost of harder debugging and only eventual consistency.
Common mistakes
- ✗Believing event-driven systems are strongly consistent — they are eventually consistent by nature
- ✗Thinking events are just logging — they are the actual communication channel between components
- ✗Assuming the producer must know which consumers exist, which would defeat the loose coupling
Follow-up questions
- →What downsides does asynchrony add compared to a synchronous call?
- →How does a consumer signal it has finished processing an event?
JuniorTheoryCommonWhy use a message broker, and what are the core entities of RabbitMQ?
Why use a message broker, and what are the core entities of RabbitMQ?
A broker decouples producers from consumers via async buffering, fan-out, and retries. In RabbitMQ a producer publishes to an exchange, which routes by routing key and bindings to one or more queues where consumers subscribe and ack messages.
Common mistakes
- ✗Saying
RabbitMQhas partitions — partitions are a Kafka concept, not an AMQP one - ✗Thinking producers publish straight to a queue, skipping the exchange and routing layer
- ✗Confusing a broker's decoupling role with an exactly-once delivery guarantee
Follow-up questions
- →How do the direct, topic, and fanout exchange types differ in routing?
- →How does partitioned offset-based consumption in the streaming platform
Kafkadiffer from RabbitMQ acks?
MiddleTheoryCommonWhat do at-least-once, at-most-once, and exactly-once delivery mean?
What do at-least-once, at-most-once, and exactly-once delivery mean?
at-most-once never retries, so a message may be lost but never duplicated. at-least-once retries on failure, so it is never lost but may arrive twice — consumers must be idempotent. exactly-once is hard end to end and is usually approximated as effectively-once via deduplication.
Common mistakes
- ✗Swapping the two — at-least-once may duplicate, at-most-once may lose; not the reverse
- ✗Believing brokers offer true end-to-end exactly-once for free without consumer deduplication
- ✗Forgetting that at-least-once is only safe when the consumer handler is idempotent
Follow-up questions
- →Which guarantee fits a payment notification and why?
- →How does consumer-side deduplication turn at-least-once into effectively-once?
MiddleTheoryCommonWhat is a dead-letter queue, and how do you reprocess failed messages?
What is a dead-letter queue, and how do you reprocess failed messages?
A dead-letter queue (DLQ) is a side queue where a broker routes messages that repeatedly fail processing or exceed a retry limit, so a poison message stops blocking the main queue. You inspect or fix the cause, then replay them — re-publishing back to the main queue or a retry queue. Consumers must be idempotent, since replay can deliver a message more than once.
Common mistakes
- ✗Thinking a DLQ discards messages rather than parking them for inspection and replay
- ✗Replaying from a DLQ without idempotent consumers, so a redelivered message double-processes
- ✗Letting a poison message retry forever in the main queue instead of routing it aside
Follow-up questions
- →What retry/backoff policy decides when a message moves to the DLQ?
- →Why must reprocessing assume at-least-once delivery and dedupe on a key?
MiddleDesignCommonDesign a durable background job queue backed by Postgres for a pool of Go workers. Producers enqueue jobs; many concurrent workers pull and run them. Requirements:
- Jobs are durable — a process crash or redeploy must not lose enqueued or in-flight work; the queue cannot live only in memory.
- Many workers claim jobs concurrently without two ever running the same job at once, and without blocking each other while contending for the next available job.
- A worker that claims a job then crashes mid-run must not strand it forever — after a visibility/lease timeout the job becomes claimable again.
- Failed jobs retry with backoff up to a maximum attempt count, after which they move to a dead-letter state instead of retrying forever.
Cover the table shape and job states, how a worker atomically claims a job, how the visibility timeout reclaims a crashed worker's job, and the retry/dead-letter flow.
Design a durable background job queue backed by Postgres for a pool of Go workers. Producers enqueue jobs; many concurrent workers pull and run them. Requirements:
- Jobs are durable — a process crash or redeploy must not lose enqueued or in-flight work; the queue cannot live only in memory.
- Many workers claim jobs concurrently without two ever running the same job at once, and without blocking each other while contending for the next available job.
- A worker that claims a job then crashes mid-run must not strand it forever — after a visibility/lease timeout the job becomes claimable again.
- Failed jobs retry with backoff up to a maximum attempt count, after which they move to a dead-letter state instead of retrying forever.
Cover the table shape and job states, how a worker atomically claims a job, how the visibility timeout reclaims a crashed worker's job, and the retry/dead-letter flow.
Store jobs in a jobs table with status, run_at, and attempts. Workers claim with SELECT ... FOR UPDATE SKIP LOCKED in a transaction, marking the row running with a lease deadline so it stays invisible until the timeout. On success delete or mark done; on failure bump attempts and reschedule run_at with backoff, moving to a dead-letter state after a max count.
Common mistakes
- ✗Using a plain
SELECTthenUPDATEwithoutSKIP LOCKED, so workers fight over the same row or block each other - ✗Holding the claim transaction open for the whole job, which ties up a DB connection and a lock for minutes
- ✗Keeping the queue only in a channel, so a crash loses every in-flight and pending job
Follow-up questions
- →How does
SKIP LOCKEDlet many workers pull different jobs without blocking? - →What reclaims a job whose worker crashed mid-run, and how is the visibility timeout enforced?
MiddleDesignCommonDesign notification fan-out for a Go service. A single domain event (e.g. "order shipped") must reach many subscribers across two channels — email and push — each via a separate external provider that can be slow or temporarily fail. Requirements:
- The HTTP request that produces the event returns immediately; provider calls must not run on the request path and add latency to the user.
- No notification is lost if the process is deployed, restarted, or crashes right after the event is produced — the event must survive in durable storage, not just memory.
- One channel or one recipient failing must not block or roll back deliveries to everyone else; each delivery retries on its own.
- Because retries mean a send can be attempted more than once, a retry must not deliver a duplicate email or push to a recipient.
Cover how the event leaves the request path, how it fans out to per-channel deliveries, and what delivery guarantee the workers assume.
Design notification fan-out for a Go service. A single domain event (e.g. "order shipped") must reach many subscribers across two channels — email and push — each via a separate external provider that can be slow or temporarily fail. Requirements: - The HTTP request that produces the event returns immediately; provider calls must not run on the request path and add latency to the user. - No notification is lost if the process is deployed, restarted, or crashes right after the event is produced — the event must survive in durable storage, not just memory. - One channel or one recipient failing must not block or roll back deliveries to everyone else; each delivery retries on its own. - Because retries mean a send can be attempted more than once, a retry must not deliver a duplicate email or push to a recipient. Cover how the event leaves the request path, how it fans out to per-channel deliveries, and what delivery guarantee the workers assume.
On the request, write one event to a durable broker or an outbox and return immediately. A consumer fans it out: it looks up subscribers, then enqueues one job per channel (email, push) so each delivery retries independently. Workers call the email and push providers, treating each send as at-least-once with idempotency keys so retries do not double-send.
Common mistakes
- ✗Calling the providers inline in a request goroutine, so a slow or failed provider loses notifications on a deploy or crash
- ✗Sending all channels in one transaction, so one failing recipient blocks or rolls back everyone else
- ✗Relying on an in-memory channel, which drops every queued notification when the process restarts
Follow-up questions
- →Why fan out to one job per channel instead of one job for the whole event?
- →How do idempotency keys stop an
at-least-onceretry from sending a duplicate email?
MiddleTheoryOccasionalHow does a consumer group in the streaming platform Kafka assign partitions, and rebalance on failure?
How does a consumer group in the streaming platform Kafka assign partitions, and rebalance on failure?
Within a consumer group each partition is assigned to exactly one consumer, so N partitions across N consumers give one partition each and maximal parallelism; extra consumers sit idle. The group coordinator triggers a rebalance when a member joins or dies, reassigning that member's partitions to the survivors — briefly pausing consumption while offsets are committed.
Common mistakes
- ✗Thinking multiple consumers in a group can read the same partition concurrently
- ✗Expecting more consumers than partitions to add throughput — the extras stay idle
- ✗Assuming a dead consumer's partitions are dropped rather than reassigned by a rebalance
Follow-up questions
- →Why does adding a 7th consumer to a 6-partition topic leave one consumer idle?
- →What consumption gap does a rebalance introduce, and how do committed offsets bound it?
MiddleTheoryOccasionalHow does the streaming platform Kafka guarantee the order of events?
How does the streaming platform Kafka guarantee the order of events?
Kafka guarantees order only within a single partition, not across a topic. Messages in one partition are appended and read in offset order; messages in different partitions have no relative order. To keep related events ordered, give them the same partition key so they hash to the same partition — at the cost of confining them to one consumer's throughput.
Common mistakes
- ✗Believing Kafka orders messages globally across a topic rather than only within a partition
- ✗Sending related events without a shared partition key, so they scatter and lose order
- ✗Expecting more partitions to preserve order — they add parallelism but break cross-partition order
Follow-up questions
- →What throughput trade-off do you accept by routing a hot key to one partition?
- →How does increasing a topic's partition count affect the ordering of existing keys?
MiddleTheoryOccasionalWhat is the outbox pattern and what consistency problem does it solve?
What is the outbox pattern and what consistency problem does it solve?
It solves the dual-write problem — a DB write and a broker publish as two non-atomic steps can leave one done, one lost. The service writes the domain change and an event row in the same transaction; a separate relay reads the outbox table and publishes.
Common mistakes
- ✗Thinking the relay publishes inside the same transaction — the relay runs separately, after commit
- ✗Believing outbox needs a distributed 2PC — its whole point is to avoid one with a single local transaction
- ✗Forgetting the relay must mark or delete published rows, or the same event ships repeatedly
Follow-up questions
- →What delivery guarantee does the relay give — at-least-once or exactly-once?
- →How would you stop the relay from publishing the same row twice?
SeniorTheoryOccasionalHow do you keep two services consistent when they use separate databases?
How do you keep two services consistent when they use separate databases?
A single ACID transaction cannot span two databases. The blocking option is 2PC, rarely used. The common option is the saga pattern — local transactions where each step has a compensating transaction undoing it on failure, yielding eventual consistency.
Common mistakes
- ✗Believing one ACID transaction can span two databases — it cannot, isolation stops at the DB boundary
- ✗Thinking a saga gives strong consistency — it gives eventual consistency with visible intermediate states
- ✗Forgetting that every saga step needs a compensating transaction for rollback on failure
Follow-up questions
- →Why is 2PC rarely used despite giving strong consistency?
- →What happens if a compensating transaction itself fails?
SeniorTheoryOccasionalHow do idempotency keys and ACK/NACK make message processing safe under retries?
How do idempotency keys and ACK/NACK make message processing safe under retries?
An idempotent handler produces the same result however many times it runs. The consumer stores each idempotency key and skips a key already seen. ACK tells the broker the message is processed; NACK or a missing ACK triggers redelivery — so at-least-once plus idempotency is safe.
Common mistakes
- ✗ACKing before processing — a crash mid-handler then loses the message because the broker won't redeliver
- ✗Thinking idempotency removes duplicate deliveries — it removes duplicate effects, the delivery still happens
- ✗Using a non-stable key like a goroutine id or timestamp instead of a stable per-message key
Follow-up questions
- →Where should the consumer store seen idempotency keys, and for how long?
- →Why must the ACK be sent only after processing fully succeeds?
