Kafka Deep Dive for Interviews

Topics, partitions, offsets, consumer groups, and delivery guarantees — explained simply, plus the operational questions interviewers ask about lag, ordering, and rebalances.

Kafka is a distributed, append-only log. That single sentence explains most of its behaviour: writes go to the end, reads are by position, and nothing is deleted just because someone read it.

The core anatomy

A topic is split into partitions. Each partition is an ordered log. Within one consumer group, each partition is read by exactly one consumer.

Term	Simple explanation
Topic	A named stream of records, e.g. `network-events`
Partition	An ordered, append-only log file. Parallelism unit — more partitions means more consumers can work at once
Offset	A record's position in a partition. Consumers track 'where am I' by committing offsets
Producer	Writes records; chooses the partition by key hash (or round-robin if no key)
Consumer group	A set of consumers sharing the work. Each partition goes to exactly one consumer in the group
Broker	A Kafka server holding partitions
Replication factor	How many copies of each partition exist across brokers (typically 3)
ISR	In-Sync Replicas — replicas fully caught up with the leader

Tip — The ordering rule — say it precisely

Kafka guarantees ordering within a partition, not across a topic. So if you need all events for one subscriber in order, use subscriber_id as the message key — the hash sends every event for that subscriber to the same partition.

Delivery guarantees

Guarantee	What it means	How you get it
At-most-once	May lose messages, never duplicates	Commit the offset before processing
At-least-once	Never loses, may duplicate	Commit the offset after processing — the common default
Exactly-once	No loss, no duplicates	Idempotent producer + transactions, or at-least-once plus an idempotent sink

Note — The senior answer on exactly-once

'True end-to-end exactly-once needs the sink to participate. In practice I run at-least-once and make the sink idempotent — a deterministic key plus MERGE into BigQuery. That is simpler to operate and gives the same business result.'

Consumer lag — the metric you live by

Lag = latest offset produced − last offset committed by the consumer. It is the single most important streaming health metric: rising lag means consumers cannot keep up and data is getting later and later.

Add consumers — but only up to the partition count; extra consumers sit idle
Add partitions — raises the parallelism ceiling (note: this changes key→partition mapping)
Make processing faster — batch the writes to the sink, drop expensive per-record work
Check for a poison message — one record that keeps failing and blocks the partition

Practice questions

QHow do you choose the number of partitions for a topic?show

Start from target throughput and required consumer parallelism, then leave headroom — because increasing partitions later breaks key-to-partition affinity.

Estimate: partitions ≈ max(target_throughput / per_consumer_throughput, desired_parallelism)
More partitions = more parallelism, but also more open file handles, more memory, longer rebalances and leader elections
Too few is the worse mistake — you cannot scale consumers beyond the partition count
Adding partitions later re-hashes keys, so existing per-key ordering is broken from that point on

Likely follow-up: What happens to ordering guarantees when you add partitions?

QWhat is a consumer group rebalance and why is it a problem?show

When a consumer joins, leaves, or times out, Kafka reassigns partitions across the group. During a classic (eager) rebalance, all consumers stop processing — a stop-the-world pause.

Common triggers: deploys, a slow consumer exceeding max.poll.interval.ms, network blips, scaling events
Fixes: use cooperative sticky assignment, tune session.timeout.ms and max.poll.records, keep per-record processing fast
A consumer that spends too long processing a batch gets kicked out — which triggers another rebalance, creating a loop

QHow does Kafka guarantee durability, and what can still lose data?show

Durability comes from replication plus the producer's acks setting; misconfiguration is what loses data.

acks=0 — fire and forget, fastest, can lose data silently
acks=1 — leader confirms only; loses data if the leader dies before replicating
acks=all + min.insync.replicas=2 — safe default for critical data
Also set enable.idempotence=true to prevent producer retries creating duplicates

QKafka vs Pub/Sub — when would you use each?show

Both are durable messaging systems, but they differ in ordering model, operations, and ecosystem.

Kafka — partition-level ordering, replay by offset, huge connector ecosystem, but you (or a vendor) operate it
Pub/Sub — fully managed, auto-scaling, no partition sizing; ordering only via ordering keys, replay via snapshots/seek
On GCP, Pub/Sub is the lower-ops default; Kafka wins when you need strict per-key ordering at scale, long retention with replay, or already have Kafka Connect pipelines

← PreviousSpark Performance Tuning and Data Skew Next →Spark Structured Streaming and Exactly-Once