Kafka Didn't Fail. Your Configuration Did.
Five real production failure patterns — consumer lag storms, broker disk overflow, schema breaking changes, hot partitions, and exactly-once misconfiguration — with the actual fixes, config tables, and monitoring setup to prevent each one.
After reading: You'll know the five failure patterns that take down Kafka pipelines, the config changes that prevent each one, and how interviewers probe Kafka knowledge in system design rounds.
Kafka runs LinkedIn's activity pipeline, Uber's trip events, and Robinhood's order stream. It processes trillions of messages a day. And yet it brings down production systems regularly — not because Kafka is unreliable, but because engineers ship it like a managed service when it is closer to a loaded weapon.
The failures are not Kafka bugs. They are configuration and operational gaps that look fine until traffic spikes, a consumer falls behind, or someone deploys a schema change on a Friday afternoon.
Where Kafka Actually Runs in Production
Kafka sits in the middle of systems that cannot afford gaps:
- Real-time fraud detection at payments companies — a 30-second lag means a fraudulent transaction clears
- Event sourcing for microservices — the event log is the source of truth, not the database
- CDC pipelines feeding data warehouses — Debezium + Kafka is the standard stack for real-time Snowflake / BigQuery ingestion
- Activity feeds at social platforms — every like, follow, and post fan-out goes through Kafka
- Log aggregation at scale — Kafka handles high-throughput log streaming where Logstash-based pipelines saturate under volume
The common thread: every downstream system trusts Kafka as the source of truth. When it misbehaves, everything downstream breaks simultaneously.
Failure #1: Consumer Lag That Never Recovers
The incident: A fintech company processing card transactions had a Kafka consumer group fall behind during a Black Friday traffic spike. Lag hit 4 million messages. The on-call engineer did what seemed logical: scaled up consumers to catch up.
The result was a rebalance storm. Every time a new consumer joined the group, Kafka triggered a full group rebalance, pausing all consumption for 30–90 seconds. More consumers made it worse. Lag kept climbing.
What actually went wrong:
max.poll.interval.ms was set to its default of 300,000ms (5 minutes). The processing logic included a downstream API call that sometimes took 6 minutes under load. Kafka marked those consumers as dead. Dead consumers leave the group. Group membership change → rebalance. Rebalance → pause. Pause → lag grows → more timeouts → more rebalances.
Watch out
The fix — three changes:
- Tune
max.poll.interval.mstoprocessing_p99_latency × 2— measure your actual processing time under load, not an estimate - Reduce
max.poll.recordsfrom 500 to 50–100 — smaller batches complete faster, avoid timeouts - Move slow downstream calls into async workers — the Kafka consumer just enqueues to an in-memory queue, a worker pool does the actual processing
Lag monitoring you need before the incident, not after:
# Prometheus alert rule
- alert: KafkaConsumerLagHigh
expr: kafka_consumer_group_lag > 10000
for: 5m
labels:
severity: warning
annotations:
summary: "Consumer group {{ $labels.group }} lag {{ $value }} on topic {{ $labels.topic }}"
Alert at 10,000 messages. By 4 million, you are already in an incident.
Failure #2: Disk Fills Up, Broker Dies
The incident: A logistics company ran Kafka on three brokers with default 7-day retention. A downstream consumer went down for 10 days during a migration. Messages accumulated. One broker's disk hit 95%, writes slowed, then the broker went offline.
With replication factor 2 and one broker down, one partition lost its leader. With only one replica remaining (the follower), if that replica was not fully caught up, the partition became unavailable. Producers started throwing NotLeaderForPartitionException. The pipeline stopped.
Key point
The math no one does before it's too late:
safe retention = min(
disk_capacity × 0.60 / daily_throughput_gb,
consumer_recovery_SLA_days
)
If you have 500 GB of disk, produce 80 GB/day, and your consumers can recover in 3 days maximum: min(500×0.6/80, 3) = min(3.75, 3) = 3 days. Set retention to 3 days. Not 7. Not 14. Not "whatever the default is."
The monitoring stack that prevents this:
The company in this incident had no disk alerting. The first signal was a producer exception in the application logs. By then, data was already unavailable.
Failure #3: Schema Change Breaks Every Consumer Overnight
The incident: A data platform team added a required Avro field to a producer schema without a default value. Deployed the producer. Every consumer running the old schema started throwing deserialization exceptions. The pipeline went silent.
No messages were lost — they were all in Kafka. Nothing was being processed. The error was not obvious because consumers were technically running. The dashboards showed consumers as healthy. Only the lag metric and downstream data freshness alerts eventually triggered.
What actually went wrong: No schema registry, no compatibility enforcement, and no deployment gate to catch breaking changes before they reached production.
Watch out
The correct schema evolution workflow:
Confluent Schema Registry setup (the minimum viable config):
# In your producer config
schema.registry.url=http://schema-registry:8081
auto.register.schemas=false
use.latest.version=true
# Compatibility mode on the registry (set per subject)
# PUT /config/your-topic-value
{ "compatibility": "BACKWARD" }
With BACKWARD compatibility enforced at the registry level, any new schema version must be readable by consumers using the previous version. The registry rejects the registration if it isn't. This catches the broken schema before it ships.
CI check to add to every producer deploy:
# Check schema compatibility before merge
curl -X POST \
-H "Content-Type: application/json" \
--data '{"schema": "'"$(cat schemas/event.avsc)"'"}' \
http://schema-registry:8081/compatibility/subjects/your-topic-value/versions/latest
# Exit 1 if response contains "is incompatible"
Failure #4: Hot Partitions Under Skewed Keys
This one doesn't generate an outage. It generates a slow death. One broker handles 80% of the traffic, its CPU pegs at 90%, consumer lag on that broker climbs steadily, and the other two brokers sit at 20%.
Why it happens: You chose a partition key that isn't uniformly distributed. user_id is usually fine. country_code with 60% of your users in India is not. is_premium_user with 5% premium users is not.
How to detect it:
# Check partition-level message rate
kafka-consumer-groups.sh \
--bootstrap-server kafka:9092 \
--describe --group your-group | \
sort -k5 -rn | head -20
If one partition's lag is 10x others, your key distribution is skewed.
Fix without rekeying: Add a salt suffix to the partition key for high-volume keys. user_id:12345 becomes user_id:12345:0 through user_id:12345:3 randomly. Fan-out the results before downstream processing. This requires application-level logic but avoids a full topic repartition.
Failure #5: Exactly-Once Semantics Configured Wrong
The classic pattern: a fintech team adds enable.idempotence=true to the producer config, assumes they have exactly-once delivery, and ships. What they actually have is idempotent producer — which prevents duplicates within a single producer session. It does not prevent duplicates if the consumer crashes after processing but before committing the offset.
For true exactly-once, you need: transactional.id on the producer, isolation.level=read_committed on the consumer, and atomic writes inside a Kafka transaction. The overhead is real — roughly 20–30% throughput reduction. Most teams are fine with at-least-once + idempotent processing logic (deduplicate by message key on the consumer side).
The Production Kafka Checklist
Before any Kafka cluster goes to production, verify every item:
Cluster configuration
- Replication factor ≥ 3 for every production topic
min.insync.replicas = 2(producer blocks if fewer than 2 replicas are in sync)unclean.leader.election.enable = false(never elect an out-of-sync replica as leader)- KRaft mode if Kafka ≥ 3.3 (ZooKeeper is deprecated)
Producer configuration
acks = all(wait for all ISR acknowledgment, not just leader)enable.idempotence = trueretries = Integer.MAX_VALUEwithdelivery.timeout.msas the actual limit- Schema registry with BACKWARD compatibility enforced
Consumer configuration
max.poll.interval.mstuned to actual processing timemax.poll.recordssized to complete within poll interval- Manual offset commit — commit after processing, not before
- Consumer lag alerting at the group level
Operations
- Disk alerting at 70% (not 95%)
- Under-replicated partition alerting at > 0
- Offline partition alerting at > 0 (this is a P0)
- Retention policy calculated from disk capacity and consumer recovery SLA
- Runbook documented and tested before the first incident
What Kafka Failure Modes Look Like in an Interview
Data engineering and ML infrastructure interviews regularly probe Kafka failure modes. The pattern is always the same: a production scenario with a hidden configuration problem. The candidate who knows the defaults is not interesting. The candidate who explains why the default breaks at scale gets the offer.
Common question framings:
- "Your Kafka consumer group fell 2 million messages behind. Walk me through your diagnosis and recovery."
- "A schema change broke your pipeline at 2 AM. How do you prevent this category of failure going forward?"
- "Design a real-time fraud detection system. What guarantees does Kafka provide and what do you have to handle yourself?"
Deep-dive: Data Engineering
Kafka, Spark, stream processing, and pipeline design — structured questions with AI-graded feedback
Frequently Asked Questions
What causes Kafka consumer lag to spike suddenly?
Consumer lag spikes when processing time per message exceeds max.poll.interval.ms, causing Kafka to mark consumers as dead and trigger group rebalances. Scaling consumers during a rebalance worsens it — each new consumer triggers another rebalance, pausing all consumption.
How do you prevent Kafka broker disk failures in production?
Calculate retention as min(disk_capacity × 0.6 / daily_throughput, consumer_recovery_SLA_days). Alert at 70% disk usage. Use replication factor 3 — replication factor 2 means one broker failure can make partitions unavailable.
What is a Kafka rebalance storm?
A rebalance storm happens when consumers continuously join and leave a group faster than Kafka can stabilize assignments. Each rebalance pauses all consumption, creating a feedback loop where lag worsens as you scale.
How do you safely evolve Kafka schemas without breaking consumers?
Use a schema registry with BACKWARD compatibility enforced at the registry level. Every new field must have a default value. Run schema compatibility checks in CI before deploying any producer change.
What is the right replication factor for Kafka in production?
Minimum replication factor 3 for production topics. Set min.insync.replicas = 2 so the producer blocks if fewer than 2 replicas are in sync. Replication factor 2 is one broker failure away from data unavailability.
What is the difference between idempotent producer and exactly-once semantics?
Idempotent producer (enable.idempotence=true) prevents duplicates within a single producer session. Exactly-once semantics (EOS) requires transactional.id on the producer, isolation.level=read_committed on the consumer, and atomic writes. EOS has 20–30% throughput overhead — most teams use at-least-once with idempotent consumer logic instead.
KnowEaseAI
Practice Kafka and Distributed Systems Interview Questions
KnowEaseAI covers Kafka failure modes, stream processing design, and real-time system design in the Data Engineering track. Real interview questions, structured answers, AI-graded practice so you know where you actually stand.
Start practicing free →Ready to practice?
Turn what you just read into skills you own
KnowEaseAI gives you AI-graded practice, structured learning checks, and a personalised roadmap. Start free — no credit card needed.
More guides
Agentic AI Engineer Interview 2026 — Complete Prep Guide
The role didn't exist 2 years ago — now every AI lab is hiring for it. What agentic AI engineers actually build, how interviews are structured, 10 real questions with model answers, and a 3-month prep plan.
MLOps Interview Questions — What's Tested in 2026
MLOps interviews now test CI/CD for ML, feature stores, model serving, and drift detection — not just pandas. Real question bank, domain-by-domain breakdown, and a 4-month roadmap from someone who's been in the loop.
Recommended reads
Guides engineers at similar stages found useful
The Tech Job Market Is Polluted, Not Broken
Data from 1,000+ LinkedIn job listings: 38–42% are staffing noise, 60% of AI Engineer JDs require nothing beyond Python, Easy Apply roles get 2.8× more applicants. The strategy shift when you understand the market is a filtering problem, not a jobs problem.
System Design Interview Guide for Indian Companies 2026
Everything you need to crack the HLD round at Swiggy, Razorpay, PhonePe, CRED, and Flipkart — company-specific bars, a full UPI payment design walkthrough, LLD round prep, anti-patterns that get candidates rejected, and SDE level calibration.
Swiggy & Zomato SDE Interview Guide 2026
A no-fluff breakdown of the Swiggy and Zomato engineering interview process — round structure, DSA patterns they favour, LLD design for Swiggy SDE-2+, real-time delivery tracking deep dive, compensation ranges, and what differentiates candidates who get offers.