Message Brokers
Message queues and Kafka — components, partition and consumer parallelism, ZooKeeper, dead-letter queues and TTL.
10 questions
JuniorTheoryVery commonWhat is a Kafka consumer group, and what happens during a rebalance?
What is a Kafka consumer group, and what happens during a rebalance?
A consumer group is a set of consumers sharing a topic's work: each partition goes to exactly one member, so each message is handled once. When a member joins or leaves, Kafka rebalances, reassigning partitions. A naive rebalance briefly stops the group; sticky assignors shrink that pause.
Common mistakes
- ✗Thinking every member of a group reads every partition
- ✗Believing two consumers in one group can share a single partition
- ✗Not knowing a rebalance can pause consumption while partitions move
Follow-up questions
- →Why can adding consumers past the partition count leave some of them idle?
- →How do frequent rebalances from a flapping consumer hurt throughput?
JuniorTheoryVery commonWhat is a message broker, and what are the core components of a queue like Kafka or RabbitMQ?
What is a message broker, and what are the core components of a queue like Kafka or RabbitMQ?
A message broker decouples producers from consumers: a producer publishes a message, the broker stores it, and a consumer reads it later — so the two need not be online at once. It also buffers load spikes for slower consumers. Core pieces are the broker (the server storing messages), producers, consumers, and the queue or topic. In Kafka a topic is split into partitions; in RabbitMQ messages route through exchanges to queues.
Common mistakes
- ✗Thinking producer and consumer must be online simultaneously
- ✗Confusing a broker with a load balancer or a plain database
- ✗Not knowing a Kafka topic is divided into partitions
Follow-up questions
- →How does decoupling producers from consumers improve resilience under load spikes?
- →How does a Kafka topic differ structurally from a RabbitMQ queue?
MiddleTheoryCommonA Kafka topic has 3 partitions and a consumer group of 4 — good or bad? What does ZooKeeper do?
A Kafka topic has 3 partitions and a consumer group of 4 — good or bad? What does ZooKeeper do?
Within one consumer group each partition is read by at most one consumer, so with 3 partitions and 4 consumers one sits idle — wasted capacity, not an error. Partitions are the unit of parallelism, so you scale by adding partitions. ZooKeeper (in older Kafka) handles cluster coordination — broker membership, controller election, metadata; newer Kafka replaces it with the built-in KRaft quorum.
Common mistakes
- ✗Thinking multiple consumers in a group can share one partition
- ✗Scaling by adding consumers past the partition count
- ✗Believing ZooKeeper stores message data rather than coordinating the cluster
Follow-up questions
- →How does the partition key decide which partition a message lands in?
- →Why does increasing partitions after the fact break per-key ordering guarantees?
SeniorTheoryCommonWhat are at-most-once, at-least-once, and exactly-once delivery, and how is exactly-once really achieved?
What are at-most-once, at-least-once, and exactly-once delivery, and how is exactly-once really achieved?
At-most-once can drop a message: commit the offset then process, and a crash loses it. At-least-once can duplicate: process, then commit, and a crash reprocesses — the pragmatic default. Exactly-once is not a broker flag: it is at-least-once plus idempotent or deduplicated processing (an idempotency key).
Common mistakes
- ✗Believing exactly-once is a broker flag rather than idempotent processing
- ✗Mixing up which commit order gives at-most-once versus at-least-once
- ✗Thinking at-least-once is unsafe rather than the pragmatic default
Follow-up questions
- →Why is committing the offset after processing the mechanism behind at-least-once?
- →How does an idempotency key let a consumer absorb a duplicate safely?
SeniorTheoryCommonHow does Kafka give durability and per-partition ordering, and what do replication and in-sync replicas (ISR) do?
How does Kafka give durability and per-partition ordering, and what do replication and in-sync replicas (ISR) do?
Each partition is replicated across brokers — one leader, the rest followers. The in-sync replicas (ISR) are followers caught up with the leader; with acks=all a write survives a broker loss once the required replicas hold it. Ordering holds only within a partition, so key related messages to one partition to keep their sequence.
Common mistakes
- ✗Assuming Kafka gives a global total order rather than per-partition order
- ✗Thinking the acks setting only tunes speed, not durability
- ✗Believing every broker holds a full copy of every partition
Follow-up questions
- →Why can a partition still lose data if acks=1 and the leader crashes before followers copy?
- →How does adding partitions later disturb the per-key ordering an existing key relied on?
JuniorTheoryOccasionalWhat is a consumer offset in Kafka, and how does a consumer resume after it restarts?
What is a consumer offset in Kafka, and how does a consumer resume after it restarts?
An offset is a message's position inside a partition — a monotonic index. A consumer group commits the offset it has processed to an internal Kafka topic, so on restart it resumes from there, not the start. With no stored offset, the auto-offset-reset policy — earliest or latest — decides where it begins.
Common mistakes
- ✗Thinking the broker marks messages read, rather than the consumer tracking an offset
- ✗Believing a restart always replays the whole partition from zero
- ✗Confusing the committed offset with a wall-clock timestamp
Follow-up questions
- →How does committing the offset before versus after processing change crash behaviour?
- →Why does each consumer group keep its own independent offset on the same partition?
JuniorTheoryOccasionalHow does a log broker like Kafka differ from a queue broker like RabbitMQ, and when do you use each?
How does a log broker like Kafka differ from a queue broker like RabbitMQ, and when do you use each?
RabbitMQ is a queue: a message goes to one competing consumer and is dropped after the ack, so work spreads across workers. Kafka is an append-only log consumers pull, each tracking its own offset, so many groups re-read one retained stream. Use a queue for task distribution, a log for streaming.
Common mistakes
- ✗Assuming Kafka deletes a message once a consumer reads it, like a queue
- ✗Thinking RabbitMQ retains messages for replay the way a log does
- ✗Treating queue-vs-log as an implementation detail, not a design choice
Follow-up questions
- →Why can a Kafka consumer replay yesterday's events while a RabbitMQ consumer cannot?
- →How do competing consumers on a queue lose the ordering a Kafka partition preserves?
MiddleDebuggingOccasionalA consumer group's lag keeps climbing and never catches up. Diagnose the cause from the group readout.
A consumer group's lag keeps climbing and never catches up. Diagnose the cause from the group readout.
Read lag per partition, not total. Partition 2 has no consumer (CONSUMER-ID empty), so its lag grows unbounded — too few consumers. Partition 1's offset barely moves while others near zero: a consumer stuck on a poison message or too slow. Fix: scale consumers to the partition count and dead-letter the bad message.
Open full question →Common mistakes
- ✗Reading only total lag and missing a single stuck or unassigned partition
- ✗Adding partitions instead of consumers when the group is under-provisioned
- ✗Assuming Kafka skips a poison message rather than retrying it forever
Follow-up questions
- →Why does a partition with no assigned consumer accumulate lag without bound?
- →How does committing the offset before processing turn a poison message into silent data loss?
MiddleTheoryRareWhat is a dead-letter queue (DLQ) in a message broker, and what is message TTL?
What is a dead-letter queue (DLQ) in a message broker, and what is message TTL?
A DLQ is a separate queue where messages land when normal delivery fails — they are rejected, exceed a retry limit, or expire — so a poison message doesn't block the main queue and can be inspected or replayed later. TTL (time to live) is a per-message or per-queue expiry: after it passes, the message is discarded or routed to the DLQ. Together they keep a queue healthy under failures.
Common mistakes
- ✗Thinking a failed message is silently deleted with no DLQ option
- ✗Confusing TTL with a minimum delay rather than an expiry
- ✗Not knowing a poison message can block the main queue without a DLQ
Follow-up questions
- →How does a max-retry/redelivery count decide when a message moves to the DLQ?
- →Why is monitoring DLQ depth a useful early warning signal in production?
SeniorDesignRareYou are designing the ingestion path for a payments platform. A Kafka topic must absorb a steady 100,000 messages per second with headroom for 2x bursts. Every event carries an account_id, and all events for one account must be processed in the exact order they arrived, while events for different accounts have no ordering relationship. A single consumer instance handles about 5,000 messages per second, and you must be able to add consumer capacity as traffic grows without reprocessing history and without breaking per-account order. Describe your scheme: how many partitions you provision and why, how you key messages, how consumer-group scaling relates to the partition count, and what happens to per-account ordering if you later raise the partition count.
You are designing the ingestion path for a payments platform. A Kafka topic must absorb a steady 100,000 messages per second with headroom for 2x bursts. Every event carries an account_id, and all events for one account must be processed in the exact order they arrived, while events for different accounts have no ordering relationship. A single consumer instance handles about 5,000 messages per second, and you must be able to add consumer capacity as traffic grows without reprocessing history and without breaking per-account order. Describe your scheme: how many partitions you provision and why, how you key messages, how consumer-group scaling relates to the partition count, and what happens to per-account ordering if you later raise the partition count.
Provision partitions for peak, not average: at ~5k msg/s per consumer, 100k needs ~20, so pick ~40 for headroom. Key each event by account_id so an account's events hit one partition, keeping order. A group scales only up to the partition count, so set it high once; raising partitions later re-hashes keys and breaks order.
Common mistakes
- ✗Sizing partitions for average load instead of peak plus headroom
- ✗Believing consumers can scale past the partition count
- ✗Not knowing raising the partition count breaks existing per-key ordering
Follow-up questions
- →Why does hashing the key to a partition fix per-account ordering but not global ordering?
- →How would you add capacity later without a repartition that reshuffles keys?
