Producer & Consumer Internals
"Call send() " and "annotate a method @KafkaListener " are one-liners. What actually happens between those calls and the broker is where most production Kafka bugs live. This module is the internals layer every later module leans on.
Learning objectives
- Beginner: State what a partition is and why message order is only guaranteed within one, not across a whole topic.
- Intermediate: Explain how a consumer's committed offset differs from what the broker itself tracks per consumer.
- Advanced: Predict how adding more consumer instances than partitions affects a consumer group's actual parallelism.
◆ The problem
If every send() call opened a fresh network round-trip to the broker, throughput would be terrible — one small record, one whole TCP request. Kafka producers need to be fast under high message volume without making the caller manage batching manually.
When you call producer.send(record), the call does not go straight to the network. It goes through these stages:
- Partitioner — decides which partition the record belongs to (by key hash, or round-robin/sticky if no key).
- Accumulator — the record is appended to an in-memory batch for that specific topic-partition, not sent yet.
- Sender thread — a dedicated background I/O thread continuously drains ready batches and sends them to the appropriate broker, independent of whatever thread called send().
- Broker append + acknowledgment — the broker appends the batch to the partition's log and responds according to the acks setting (see Module 13).
send() returns almost immediately — it hands the record to the accumulator and a background sender thread does the actual network I/O. This is why the calling thread (e.g. your REST controller thread) isn't blocked on network latency by default.
◆ Under the hood
Batching & buffering are governed by two settings that trade latency for throughput: batch.size (max bytes per batch before it's sent) and linger.ms (how long to wait for a batch to fill before sending it anyway, even if not full). linger.ms=0 (the default) sends almost immediately once anything is queued; setting it to e.g. 10–20ms lets more records accumulate into fewer, larger, more efficient batches — a small, deliberate latency cost for meaningfully higher throughput under load.
▲ Pitfall
Because sending is asynchronous by default, an exception during the actual broker round-trip does not throw back into your calling code unless you attach a callback or block on the returned Future. Fire-and-forget sends can silently drop failed records if you never check the result — see the async vs. sync error handling comparison in Module 13.
A consumer's core loop is deceptively simple to describe and easy to misunderstand: it repeatedly calls poll(Duration), which does several things in one call — sends fetch requests to the leaders of all partitions it's currently assigned, returns any records already buffered from prior fetches, sends periodic heartbeats to the group coordinator, and processes any pending rebalance if the group membership changed. Nothing happens between poll() calls — if your processing logic takes too long, the next poll() is late, heartbeats fall behind schedule, and the group coordinator can decide this consumer is dead (see rebalancing below).
An offset is scoped to a specific (consumer group, topic, partition) triple — group A and group B reading the same topic track completely independent offsets. Kafka stores committed offsets durably in an internal topic, __consumer_offsets, which is itself just a regular (compacted) Kafka topic — offset storage uses the exact same durability mechanism as your business data.
This is what makes replay possible: because records aren't deleted on read, and offsets are just a pointer a consumer can choose to reset, a new consumer group (or a manually reset offset) can re-read the entire retained history of a topic from the beginning.
Within a consumer group, Kafka guarantees each partition is assigned to exactly one consumer instance at a time. This is the whole mechanism behind horizontal scaling of consumption:
| Consumers in group vs. partitions | What happens |
|---|---|
| Consumers < partitions | Some consumers own more than one partition — still works, just less parallel. |
| Consumers = partitions | Perfectly balanced: one partition per consumer. |
| Consumers > partitions | Extra consumers sit idle — a partition can't be split between two consumers in the same group. |
◆ Under the hood — what happens during a rebalance
A rebalance is triggered when a consumer joins the group, leaves it (gracefully or by crashing), or is declared dead because it missed too many heartbeats (governed by session.timeout.ms, covered in Module 15). The group coordinator broadcasts a rebalance, every member stops consuming, partitions are reassigned among the (possibly changed) set of members, and consumption resumes. Any record a consumer had already fetched and started processing, but not yet committed the offset for, is at risk of being re-delivered to whichever consumer now owns that partition — this is exactly why Kafka's default delivery guarantee is at-least-once, not exactly-once, unless you add idempotency/transactions on top (Modules 13 & 16).
Physically, a partition is a sequence of segment files on disk. Kafka only ever appends to the active (newest) segment; older, closed segments are immutable. Retention (deciding when to delete old data) operates at the segment level, not the individual-record level — a whole segment is deleted once every record in it is older than retention.ms (time-based) or the partition exceeds retention.bytes (size-based), whichever is configured.
▲ Pitfall
Retention is completely independent of whether any consumer has read a record. A slow or offline consumer group does not pause deletion — if it falls behind retention, it silently loses access to the oldest unread records. Monitor consumer lag against retention window, not just "is the consumer running."
✓ Quick recap
Why does send() return almost instantly even though the broker hasn't necessarily confirmed the write? The record is handed to an in-memory accumulator and sent asynchronously by a background sender thread. What single call does most of a consumer's real work? poll() — fetching, heartbeating, and rebalance handling all happen inside it. What can happen to an already-fetched-but-uncommitted record during a rebalance? It can be redelivered to a different consumer that now owns the partition — the basis of at-least-once delivery. Does retention wait for consumers to catch up? No — retention deletes old segments purely by age/size, regardless of consumer lag.
Want a visual for this concept?
Generate a diagram tailored to “Producer & Consumer Internals” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →