Producer Reliability & Errors
Every setting in this module trades something for something. Understanding the trade, not just the property name, is what separates a config you copy-pasted from one you can defend in a design review.
Learning objectives
- Beginner: Since Kafka 3.0 / Spring Boot 3.x, the producer defaults to acks=all with idempotence enabled — a deliberate change from Kafka's older, weaker acks=1 default; know what you'd be opting INTO by weakening it, not just what the default is.
- Intermediate: acks=all + min.insync.replicas=2 for a service where losing an order confirmation is a real business problem.
- Advanced: Full recommended config (§13.6) combined with Module 16's transactional publish, so a batch of related events is both duplicate-safe (idempotence) and atomic (transactions) — the two mechanisms solve different problems and compose.
◆ The problem
The producer's send() returns before the broker necessarily has durable, replicated proof of the write — how much confirmation you demand before calling a write "successful" is a tunable trade-off, not a fixed guarantee Kafka makes for you.
| acks | What it waits for | Durability | Latency |
|---|---|---|---|
| 0 | Nothing — fire and forget | None — silent data loss on broker failure is possible | Lowest |
| 1 | Leader broker only | Lost if the leader fails before followers replicate | Medium |
| all / -1 | All current ISR members | Survives leader failure, if min.insync.replicas is also set sensibly | Highest |
◆ Under the hood — why acks=all alone isn't enough
acks=all means "wait for all current ISR members," but if the ISR has shrunk to just the leader (every follower fell behind or is down), "all ISR members" is satisfied by the leader alone — functionally the same durability as acks=1. min.insync.replicas closes this gap: it makes the write fail outright (rather than silently downgrade) if fewer than that many ISR members are available, so the producer at least finds out its durability requirement couldn't be met.
▲ Pitfall
See also Module 02 §4's pitfall: setting min.insync.replicas equal to the full replication factor makes the topic unwritable the instant any single broker is down — there's no ISR member to spare.
retries controls how many times the producer automatically retries a send that failed for a retriable reason (e.g. a transient NotLeaderForPartitionException during a leader election). retry.backoff.ms spaces retries apart instead of hammering the broker immediately, giving a transient condition (like an in-progress leader election) time to resolve.
◆ The problem
Retries solve "the send failed, try again" — but what if the send actually succeeded and only the broker's acknowledgment was lost in transit? A naive retry now produces a duplicate record, even though nothing was actually wrong with the first write.
An idempotent producer (enable.idempotence=true) assigns each producer instance a unique Producer ID and tags every record with a per-partition sequence number. The broker tracks the last sequence number it committed per producer/partition and silently discards a retry carrying a sequence number it's already seen — turning "retry might duplicate" into "retry is safe."
spring: kafka: producer: properties: enable.idempotence: true acks: all
◆ Under the hood
Enabling idempotence automatically forces acks=all and retries > 0 (the client refuses an inconsistent combination) — idempotence is only meaningful alongside retries and full acknowledgment; it has nothing to protect against otherwise.
💻 Code example
spring: kafka: producer: properties: enable.idempotence: true acks: all
This caps how many produce requests can be outstanding (sent but not yet acknowledged) on one connection at once. It matters specifically alongside retries: with multiple in-flight requests and a retry in the mix, a later batch can in principle be acknowledged before an earlier, retried one, silently reordering records.
| Setting | Effect on ordering with retries enabled |
|---|---|
| max.in.flight.requests.per.connection=1 | Ordering fully preserved, at some throughput cost |
| >1 without idempotence | Retries can reorder records |
| >1 with idempotence enabled (up to 5) | Ordering preserved — the broker uses sequence numbers to reject/reorder duplicates correctly even with several in-flight requests |
| Property | Governs |
|---|---|
| request.timeout.ms | How long the client waits for a single broker response before considering that request failed. |
| delivery.timeout.ms | The total upper bound across the batch's time in the accumulator, all retries, and all requests — must be ≥ linger.ms + request.timeout.ms, or the client refuses to start. |
| Property | Value | Why |
|---|---|---|
| acks | all | Wait for full ISR acknowledgment, not just the leader. |
| min.insync.replicas | 2 (with RF=3) | Fail loudly rather than silently accept leader-only durability when the ISR shrinks. |
| enable.idempotence | true | Retries can't create duplicates. |
| retries | high (e.g. Integer.MAX_VALUE, bounded by delivery.timeout.ms) | Let the timeout — not an arbitrary retry count — decide when to give up. |
| max.in.flight.requests.per.connection | 5 | Safe with idempotence enabled; keeps throughput without sacrificing order. |
Revisit §03.1's pitfall: async sends need an explicit callback or you silently lose failures. This is the concrete pattern for both paths:
private void handleFailure(String value, Throwable ex) { if (ex instanceof RetriableException) { log.warn("retriable failure, Kafka client will retry: {}", ex.getMessage()); } else { log.error("non-retriable failure publishing: {}", value, ex); // non-retriable: surface to caller / alert — retrying won't help } }
For the sync path (§06.3), the failure surfaces directly as a thrown ExecutionException at the.get() call site — simpler to reason about, at the cost of blocking the calling thread for up to the timeout on every send.
✓ Quick recap
Why isn't acks=all sufficient durability on its own? "All ISR members" can shrink to just the leader; min.insync.replicas is what forces the write to fail rather than silently accept leader-only durability. What specific failure mode does the idempotent producer prevent that plain retries don't? Duplicate records from a retry after a successful send whose acknowledgment was lost in transit. How can retries reorder records, and what fixes it? With max.in.flight.requests.per.connection > 1 and no idempotence, a retried earlier batch can be acknowledged after a later one; idempotence (up to 5 in-flight) preserves order safely.
💻 Code example
private void handleFailure(String value, Throwable ex) { if (ex instanceof RetriableException) { log.warn("retriable failure, Kafka client will retry: {}", ex.getMessage()); } else { log.error("non-retriable failure publishing: {}", value, ex); // non-retriable: surface to caller / alert — retrying won't help } }
Want a visual for this concept?
Generate a diagram tailored to “Producer Reliability & Errors” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →