Message Brokers
A message broker is an intermediary that decouples the sender of data from its processor. A producer publishes a message and moves on, the broker stores it durably, and a consumer picks it up later — the two sides need not be online at once or know about each other. This decoupling in time and space is what makes the system resilient to load spikes and to individual service failures.
The topic gathers three traps people reliably stumble on. First, treating a broker as a load balancer or a plain database: it specifically stores and routes messages between decoupled services. Second, thinking several consumers in one group share a partition: parallelism in Kafka is bounded by the number of partitions, not consumers. Third, not knowing about the dead-letter queue and time to live: without them a failed message is either lost silently or blocks the whole queue. The breakdown is in the layers below.
Topic map
- Message broker basics — decoupling producer from consumer and the core components of a queue in Kafka and RabbitMQ.
- Kafka partitions and consumer groups — the partition as the unit of parallelism, the "one partition ↔ one consumer per group" rule, and the role of ZooKeeper/KRaft.
- DLQ and message TTL — where a failed message goes and how time to live protects the queue from poison messages.
Common traps
| Mistake | Consequence |
|---|---|
| Assuming producer and consumer must be online together | Loses the whole point of asynchronous decoupling; the system is fragile to failures |
| Confusing a broker with a load balancer or a database | Wrong model — a broker specifically stores and routes messages |
| Thinking several consumers in a group read one partition | Extra consumers sit idle; parallelism does not grow |
| Scaling by adding consumers past the partition count | Capacity is bounded by partitions, not consumers |
| Believing ZooKeeper stores message data | ZooKeeper coordinates the cluster (KRaft in newer Kafka); data lives in the partition logs |
| Not knowing about the DLQ | A poison message blocks the queue or is lost silently |
| Confusing TTL with a minimum delay | TTL is an expiry, not a delay before delivery |
Interview relevance
Brokers are asked to check whether you separate asynchronous decoupling from a synchronous call and understand exactly where the parallelism lives. A candidate who says "a partition is the unit of parallelism, so you scale by partitions, and a consumer group reads each partition with at most one consumer" immediately gets ahead of "I'll add consumers and it'll go faster".
Typical checks:
- Why a broker exists and what its components are — producer, consumer, the broker itself, the queue/topic.
- Working the "3 partitions, 4 consumers" question — one sits idle, that's wasted capacity, not an error.
- What ZooKeeper does and what replaces it in newer Kafka.
- What a DLQ and TTL are and how they keep the queue healthy under failures.
Common wrong answer: "to speed up processing I'll add consumers to the group". Past the partition count the extra consumers sit idle: the partition count sets the scale. And "a failed message is just deleted" — without a DLQ it is either lost or stuck, blocking the queue.