Skip to main content
🪔✨10% off ProKNOWEASEAI10Claim offer →✨🎆
◆◆◆ Advanced

Kafka Didn't Fail. Your Configuration Did.

Five real production failure patterns — consumer lag storms, broker disk overflow, schema breaking changes, hot partitions, and exactly-once misconfiguration — with the actual fixes, config tables, and monitoring setup to prevent each one.

After reading: You'll know the five failure patterns that take down Kafka pipelines, the config changes that prevent each one, and how interviewers probe Kafka knowledge in system design rounds.

18 min read··By KnowEaseAI Editorial
Liked by 142 engineers
Share:XLinkedIn

Kafka runs LinkedIn's activity pipeline, Uber's trip events, and Robinhood's order stream. It processes trillions of messages a day. And yet it brings down production systems regularly — not because Kafka is unreliable, but because engineers ship it like a managed service when it is closer to a loaded weapon.

The failures are not Kafka bugs. They are configuration and operational gaps that look fine until traffic spikes, a consumer falls behind, or someone deploys a schema change on a Friday afternoon.


Where Kafka Actually Runs in Production

Kafka sits in the middle of systems that cannot afford gaps:

  • Real-time fraud detection at payments companies — a 30-second lag means a fraudulent transaction clears
  • Event sourcing for microservices — the event log is the source of truth, not the database
  • CDC pipelines feeding data warehouses — Debezium + Kafka is the standard stack for real-time Snowflake / BigQuery ingestion
  • Activity feeds at social platforms — every like, follow, and post fan-out goes through Kafka
  • Log aggregation at scale — Kafka handles high-throughput log streaming where Logstash-based pipelines saturate under volume

The common thread: every downstream system trusts Kafka as the source of truth. When it misbehaves, everything downstream breaks simultaneously.


Failure #1: Consumer Lag That Never Recovers

The incident: A fintech company processing card transactions had a Kafka consumer group fall behind during a Black Friday traffic spike. Lag hit 4 million messages. The on-call engineer did what seemed logical: scaled up consumers to catch up.

The result was a rebalance storm. Every time a new consumer joined the group, Kafka triggered a full group rebalance, pausing all consumption for 30–90 seconds. More consumers made it worse. Lag kept climbing.

What actually went wrong:

max.poll.interval.ms was set to its default of 300,000ms (5 minutes). The processing logic included a downstream API call that sometimes took 6 minutes under load. Kafka marked those consumers as dead. Dead consumers leave the group. Group membership change → rebalance. Rebalance → pause. Pause → lag grows → more timeouts → more rebalances.

Watch out

Never scale Kafka consumers during a rebalance storm. Pause new deployments, let the group stabilize, then scale. Adding consumers mid-storm extends the storm.

The fix — three changes:

  1. Tune max.poll.interval.ms to processing_p99_latency × 2 — measure your actual processing time under load, not an estimate
  2. Reduce max.poll.records from 500 to 50–100 — smaller batches complete faster, avoid timeouts
  3. Move slow downstream calls into async workers — the Kafka consumer just enqueues to an in-memory queue, a worker pool does the actual processing

Lag monitoring you need before the incident, not after:

YAML
# Prometheus alert rule
- alert: KafkaConsumerLagHigh
  expr: kafka_consumer_group_lag > 10000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Consumer group {{ $labels.group }} lag {{ $value }} on topic {{ $labels.topic }}"

Alert at 10,000 messages. By 4 million, you are already in an incident.


Failure #2: Disk Fills Up, Broker Dies

The incident: A logistics company ran Kafka on three brokers with default 7-day retention. A downstream consumer went down for 10 days during a migration. Messages accumulated. One broker's disk hit 95%, writes slowed, then the broker went offline.

With replication factor 2 and one broker down, one partition lost its leader. With only one replica remaining (the follower), if that replica was not fully caught up, the partition became unavailable. Producers started throwing NotLeaderForPartitionException. The pipeline stopped.

Key point

Replication factor 2 is not redundancy. It is one broker failure away from data unavailability. Use replication factor 3 for every production topic.

The math no one does before it's too late:

safe retention = min(
  disk_capacity × 0.60 / daily_throughput_gb,
  consumer_recovery_SLA_days
)

If you have 500 GB of disk, produce 80 GB/day, and your consumers can recover in 3 days maximum: min(500×0.6/80, 3) = min(3.75, 3) = 3 days. Set retention to 3 days. Not 7. Not 14. Not "whatever the default is."

The monitoring stack that prevents this:

The company in this incident had no disk alerting. The first signal was a producer exception in the application logs. By then, data was already unavailable.


Failure #3: Schema Change Breaks Every Consumer Overnight

The incident: A data platform team added a required Avro field to a producer schema without a default value. Deployed the producer. Every consumer running the old schema started throwing deserialization exceptions. The pipeline went silent.

No messages were lost — they were all in Kafka. Nothing was being processed. The error was not obvious because consumers were technically running. The dashboards showed consumers as healthy. Only the lag metric and downstream data freshness alerts eventually triggered.

What actually went wrong: No schema registry, no compatibility enforcement, and no deployment gate to catch breaking changes before they reached production.

Watch out

Treat schema changes like database migrations. Reviewed, versioned, and rollback-tested before they touch production. A new required field without a default is a breaking change — full stop.

The correct schema evolution workflow:

Confluent Schema Registry setup (the minimum viable config):

properties
# In your producer config
schema.registry.url=http://schema-registry:8081
auto.register.schemas=false
use.latest.version=true

# Compatibility mode on the registry (set per subject)
# PUT /config/your-topic-value
{ "compatibility": "BACKWARD" }

With BACKWARD compatibility enforced at the registry level, any new schema version must be readable by consumers using the previous version. The registry rejects the registration if it isn't. This catches the broken schema before it ships.

CI check to add to every producer deploy:

Bash
# Check schema compatibility before merge
curl -X POST \
  -H "Content-Type: application/json" \
  --data '{"schema": "'"$(cat schemas/event.avsc)"'"}' \
  http://schema-registry:8081/compatibility/subjects/your-topic-value/versions/latest

# Exit 1 if response contains "is incompatible"

Failure #4: Hot Partitions Under Skewed Keys

This one doesn't generate an outage. It generates a slow death. One broker handles 80% of the traffic, its CPU pegs at 90%, consumer lag on that broker climbs steadily, and the other two brokers sit at 20%.

Why it happens: You chose a partition key that isn't uniformly distributed. user_id is usually fine. country_code with 60% of your users in India is not. is_premium_user with 5% premium users is not.

How to detect it:

Bash
# Check partition-level message rate
kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --describe --group your-group | \
  sort -k5 -rn | head -20

If one partition's lag is 10x others, your key distribution is skewed.

Fix without rekeying: Add a salt suffix to the partition key for high-volume keys. user_id:12345 becomes user_id:12345:0 through user_id:12345:3 randomly. Fan-out the results before downstream processing. This requires application-level logic but avoids a full topic repartition.


Failure #5: Exactly-Once Semantics Configured Wrong

The classic pattern: a fintech team adds enable.idempotence=true to the producer config, assumes they have exactly-once delivery, and ships. What they actually have is idempotent producer — which prevents duplicates within a single producer session. It does not prevent duplicates if the consumer crashes after processing but before committing the offset.

For true exactly-once, you need: transactional.id on the producer, isolation.level=read_committed on the consumer, and atomic writes inside a Kafka transaction. The overhead is real — roughly 20–30% throughput reduction. Most teams are fine with at-least-once + idempotent processing logic (deduplicate by message key on the consumer side).


The Production Kafka Checklist

Before any Kafka cluster goes to production, verify every item:

Cluster configuration

  • Replication factor ≥ 3 for every production topic
  • min.insync.replicas = 2 (producer blocks if fewer than 2 replicas are in sync)
  • unclean.leader.election.enable = false (never elect an out-of-sync replica as leader)
  • KRaft mode if Kafka ≥ 3.3 (ZooKeeper is deprecated)

Producer configuration

  • acks = all (wait for all ISR acknowledgment, not just leader)
  • enable.idempotence = true
  • retries = Integer.MAX_VALUE with delivery.timeout.ms as the actual limit
  • Schema registry with BACKWARD compatibility enforced

Consumer configuration

  • max.poll.interval.ms tuned to actual processing time
  • max.poll.records sized to complete within poll interval
  • Manual offset commit — commit after processing, not before
  • Consumer lag alerting at the group level

Operations

  • Disk alerting at 70% (not 95%)
  • Under-replicated partition alerting at > 0
  • Offline partition alerting at > 0 (this is a P0)
  • Retention policy calculated from disk capacity and consumer recovery SLA
  • Runbook documented and tested before the first incident

What Kafka Failure Modes Look Like in an Interview

Data engineering and ML infrastructure interviews regularly probe Kafka failure modes. The pattern is always the same: a production scenario with a hidden configuration problem. The candidate who knows the defaults is not interesting. The candidate who explains why the default breaks at scale gets the offer.

Common question framings:

  • "Your Kafka consumer group fell 2 million messages behind. Walk me through your diagnosis and recovery."
  • "A schema change broke your pipeline at 2 AM. How do you prevent this category of failure going forward?"
  • "Design a real-time fraud detection system. What guarantees does Kafka provide and what do you have to handle yourself?"

Deep-dive: Data Engineering

Kafka, Spark, stream processing, and pipeline design — structured questions with AI-graded feedback

Study →

Frequently Asked Questions

What causes Kafka consumer lag to spike suddenly?

Consumer lag spikes when processing time per message exceeds max.poll.interval.ms, causing Kafka to mark consumers as dead and trigger group rebalances. Scaling consumers during a rebalance worsens it — each new consumer triggers another rebalance, pausing all consumption.

How do you prevent Kafka broker disk failures in production?

Calculate retention as min(disk_capacity × 0.6 / daily_throughput, consumer_recovery_SLA_days). Alert at 70% disk usage. Use replication factor 3 — replication factor 2 means one broker failure can make partitions unavailable.

What is a Kafka rebalance storm?

A rebalance storm happens when consumers continuously join and leave a group faster than Kafka can stabilize assignments. Each rebalance pauses all consumption, creating a feedback loop where lag worsens as you scale.

How do you safely evolve Kafka schemas without breaking consumers?

Use a schema registry with BACKWARD compatibility enforced at the registry level. Every new field must have a default value. Run schema compatibility checks in CI before deploying any producer change.

What is the right replication factor for Kafka in production?

Minimum replication factor 3 for production topics. Set min.insync.replicas = 2 so the producer blocks if fewer than 2 replicas are in sync. Replication factor 2 is one broker failure away from data unavailability.

What is the difference between idempotent producer and exactly-once semantics?

Idempotent producer (enable.idempotence=true) prevents duplicates within a single producer session. Exactly-once semantics (EOS) requires transactional.id on the producer, isolation.level=read_committed on the consumer, and atomic writes. EOS has 20–30% throughput overhead — most teams use at-least-once with idempotent consumer logic instead.

KnowEaseAI

Practice Kafka and Distributed Systems Interview Questions

KnowEaseAI covers Kafka failure modes, stream processing design, and real-time system design in the Data Engineering track. Real interview questions, structured answers, AI-graded practice so you know where you actually stand.

Start practicing free →
Was this guide helpful?
Share:XLinkedIn

Ready to practice?

Turn what you just read into skills you own

KnowEaseAI gives you AI-graded practice, structured learning checks, and a personalised roadmap. Start free — no credit card needed.