Consumer Timing Configurations
Most "mystery rebalance storms" in production trace back to one of the three settings on this page being wrong relative to how long your listener actually takes to run.
Learning objectives
- Beginner: Name the three categories of consumer timing configuration (broker-side, fetch-related, connection-related).
- Intermediate: Explain what max.poll.interval.ms actually protects against and what happens when it's exceeded.
- Advanced: Diagnose a "mystery rebalance storm" in production by tracing it back to a specific misconfigured timing setting.
These live on the topic/broker itself, not on the consumer client — worth knowing about here even though they're a different category from this module's main subject (consumer-side timing configs below): retention.ms (§03.5) determines how far back a reset-to-earliest consumer can actually go, and log.segment.ms influences how quickly older segments become eligible for deletion once they age past retention.
| Property | Effect | Trade-off |
|---|---|---|
| fetch.min.bytes | Broker won't respond to a fetch until at least this many bytes are available (or fetch.max.wait.ms elapses) | Higher = fewer, larger fetch responses (throughput); adds latency for low-volume topics |
| fetch.max.wait.ms | Max time the broker holds a fetch request open waiting to satisfy fetch.min.bytes | Upper bound on the latency traded above |
| max.poll.records | Max records returned per poll() call | Higher = fewer poll cycles, but each cycle's processing takes longer — interacts directly with §15.3 below |
◆ The problem
A consumer that's alive and healthy can still get kicked out of its group and trigger a rebalance, purely because it took too long between poll() calls — Kafka has no way to distinguish "this consumer died" from "this consumer is just slow," so it uses timeouts as a proxy, and a wrong timeout produces a false positive.
| Property | What it governs |
|---|---|
| session.timeout.ms | How long the group coordinator waits without a heartbeat before declaring the consumer dead and triggering a rebalance. |
| heartbeat.interval.ms | How often the consumer's background heartbeat thread pings the coordinator — should be well under session.timeout.ms (commonly ~1/3). |
| max.poll.interval.ms | The maximum time allowed between calls to poll() before the consumer is presumed stuck/dead, independent of heartbeats (heartbeats run on a separate thread in modern Kafka clients). |
◆ Under the hood
Since Kafka 0.10.1+, heartbeats run on a background thread separate from the thread calling poll() /processing records — so a slow listener method doesn't necessarily stop heartbeats. What it does risk is exceeding max.poll.interval.ms: if your listener logic takes longer than this between poll() calls (e.g. because it's synchronously calling a slow downstream service per record), the coordinator treats the consumer as stuck and forces a rebalance — even though it was heartbeating the entire time.
▲ Pitfall
A common real-world cause of "constant rebalancing" is max.poll.records set too high combined with slow per-record processing (e.g. a synchronous external API call per record) and a default max.poll.interval.ms — the batch simply takes longer to process than the interval allows. Fix by lowering max.poll.records, speeding up per-record processing, or raising max.poll.interval.ms deliberately — not by blindly increasing timeouts everywhere.
✓ Quick recap
What's the practical difference between session.timeout.ms and max.poll.interval.ms? session.timeout.ms tracks missed heartbeats (background thread); max.poll.interval.ms tracks how long between actual poll() calls on the processing thread — a slow listener can trip the latter without missing a single heartbeat. What's a common root cause of "mystery" rebalance storms? max.poll.records set too high combined with slow per-record processing, exceeding max.poll.interval.ms even though heartbeats never stopped.
One layer below the group-membership timeouts above sits the raw TCP connection between your consumer and the broker — a separate set of settings governs that connection specifically, independent of group membership:
| Property | What it governs |
|---|---|
| request.timeout.ms | How long the client waits for a response to any single request (fetch, metadata, etc.) before treating it as failed and retrying. |
| connection.max.idle.ms | How long an idle connection to a broker is kept open before the client proactively closes it — closed connections get transparently re-established on the next request. |
| reconnect.backoff.ms | Base wait time before retrying a failed connection attempt to a broker. |
| reconnect.backoff.max.ms | The ceiling that backoff grows to under repeated failures (via exponential backoff) — prevents a client from either hammering a down broker or waiting unboundedly long once it recovers. |
These are a genuinely separate concern from session.timeout.ms/heartbeat.interval.ms/max.poll.interval.ms above: those three govern whether the CONSUMER GROUP considers you still a member; these four govern whether your CLIENT's underlying TCP connection to a broker is healthy and how it recovers when it isn't. A consumer can be correctly configured on the group-membership side and still behave badly under a flaky network if these are left at defaults that don't suit your environment.
Want a visual for this concept?
Generate a diagram tailored to “Consumer Timing Configurations” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →