A consumer group's lag keeps climbing and never catches up. Diagnose the cause from the group readout.
An order-processor consumer group reads a 3-partition orders topic. Alerts fire: end-to-end latency is rising and lag keeps growing. You run the describe command and get:
$ kafka-consumer-groups.sh --bootstrap-server broker:9092 \
--group order-processor --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
order-processor orders 0 4182233 4182240 7 consumer-1
order-processor orders 1 3120004 5987410 2867406 consumer-2
order-processor orders 2 - 6011882 6011882 -Explain why the lag grows and give the fix.
Read lag per partition, not total. Partition 2 has no consumer (CONSUMER-ID empty), so its lag grows unbounded — too few consumers. Partition 1's offset barely moves while others near zero: a consumer stuck on a poison message or too slow. Fix: scale consumers to the partition count and dead-letter the bad message.
- ✗Reading only total lag and missing a single stuck or unassigned partition
- ✗Adding partitions instead of consumers when the group is under-provisioned
- ✗Assuming Kafka skips a poison message rather than retrying it forever
- →Why does a partition with no assigned consumer accumulate lag without bound?
- →How does committing the offset before processing turn a poison message into silent data loss?
Read lag per partition, not in aggregate — the three partitions fail differently:
- Partition 2 —
CONSUMER-IDis empty andCURRENT-OFFSETis-: nothing reads it. The group has fewer consumers than partitions, so its lag (6,011,882) grows without bound. - Partition 1 —
consumer-2is assigned butCURRENT-OFFSETbarely moves at a lag of 2,867,406: the consumer is stuck. Usually a poison message it keeps failing and retrying, or simply a consumer that is too slow. - Partition 0 — healthy (lag 7).
Fix — add consumers up to the partition count and route the bad message to a DLQ:
# 2 consumers on 3 partitions → bring it up to 3
kubectl scale deployment/order-processor --replicas=3
# after the rebalance each partition gets its own consumer;
# the poison message moves to a dead-letter topic after N retries,
# so partition 1 drains againDo not add partitions to fix this — that re-hashes keys and breaks per-key ordering.
