intermediate~3h

Consumer Errors, Retry & Recovery

A producer failure is your problem alone. A consumer failure can block an entire partition behind one bad record forever, if you don't design for it. This is the most consequential module on the site for production stability.

Learning objectives

Beginner: No custom error handler — acceptable only in a throwaway prototype, since any bad record permanently blocks its partition.
Intermediate: DefaultErrorHandler with exponential backoff and a Dead Letter Topic recoverer, classifying at least one known non-retryable exception type.
Advanced: Hybrid recovery (DLT + DB) with a RetryListener feeding metrics, combined with idempotent persistence (Module 10) so a manually replayed DLT record can never double-apply its effect.

◆ The problem

Your @KafkaListener method throws an exception — a downstream database is briefly down, or a record is malformed in a way your code doesn't handle. What happens to that record, and to the partition it came from?

A truly bare Kafka consumer, with no framework at all, would behave badly here: catch nothing, never advance the offset, and re-fetch the same record on every poll() forever — the partition genuinely stuck, blocking every record behind it (order within a partition is strict; nothing can jump the queue).

Spring Kafka doesn't leave you at that starting point. Out of the box, with zero custom configuration, Spring Boot auto-configures a DefaultErrorHandler on every listener container: it retries a failed record a bounded number of times (roughly 10 attempts, spaced apart) and, once retries are exhausted, logs the failure and lets the offset commit anyway — the partition moves on, it does not get stuck. That default is genuinely reasonable for a demo, but it silently loses the failed record (nothing captures it anywhere), which is exactly the gap §14.6's recovery strategies (DLT, DB, or both) exist to close.

DefaultErrorHandler is already active on every listener container by default — the beginner mistake is thinking you need to define it just to get retry behavior at all. What you're actually doing by declaring your own @Bean isn't turning retries on, it's replacing the default backoff/recoverer with ones you control: a specific BackOff strategy instead of the default one, and a specific recoverer (§14.6) instead of Spring's default log-and-skip.

DefaultErrorHandler's decision flow, with or without your own customization: classify the exception, retry with backoff if retryable, and hand off to a recoverer once retries are exhausted (or immediately, for a non-retryable exception).

@Bean
public DefaultErrorHandler errorHandler() {
    ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
    backOff.setMaxInterval(10_000L);
    backOff.setMaxElapsedTime(60_000L);

    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer(), backOff);
    handler.addNotRetryableExceptions(IllegalArgumentException.class); // see §14.5
    return handler;
}

💻 Code example

@Bean
public DefaultErrorHandler errorHandler() {
    ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
    backOff.setMaxInterval(10_000L);
    backOff.setMaxElapsedTime(60_000L);

    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer(), backOff);
    handler.addNotRetryableExceptions(IllegalArgumentException.class); // see §14.5
    return handler;
}

Fixed-interval retry hammers a struggling downstream dependency at a constant rate; exponential backoff (1s, 2s, 4s, 8s...) gives it progressively more room to recover, capped by maxInterval so retries don't grow unbounded, and bounded overall by maxElapsedTime so retrying eventually gives up and moves to recovery rather than continuing forever.

A RetryListener is a pure observability hook — it doesn't change retry behavior, it lets you log/measure each attempt, which matters operationally: without it, you only see the final success or the final recovery, with no visibility into how many attempts a record actually needed.

handler.setRetryListeners((record, ex, deliveryAttempt) ->
    log.warn("retry attempt {} for partition={} offset={}: {}",
        deliveryAttempt, record.partition(), record.offset(), ex.getMessage()));

💻 Code example

handler.setRetryListeners((record, ex, deliveryAttempt) ->
    log.warn("retry attempt {} for partition={} offset={}: {}",
        deliveryAttempt, record.partition(), record.offset(), ex.getMessage()));

Not every failure deserves a retry. A transient DataAccessResourceFailureException (DB briefly unreachable) is worth retrying — the same call might succeed seconds later. An IllegalArgumentException from a structurally invalid record will fail identically on every retry — retrying it just delays reaching recovery for no benefit, and wastes the backoff window.

Exception type	Typical classification	Why
DataAccessResourceFailureException	Retryable	Transient infrastructure issue, likely to self-resolve
IllegalArgumentException (bad payload shape)	Non-retryable	Deterministic failure — will fail the same way every time
OptimisticLockingFailureException	Often retryable	A concurrent update collision that a fresh attempt may not hit again

Once retries are exhausted (or an exception is classified non-retryable), a recoverer decides what happens to the record so the partition can move past it. Spring's own built-in default recoverer just logs and skips — silent, no record of the failure anywhere. These three strategies replace that default with something that actually captures the failure, in increasing order of operational sophistication:

Strategy	What it does	Good for	Weakness
1. Dead Letter Topic	Republishes the failed record to a separate library-events.DLT topic	Reprocessing later by a separate consumer/tooling; keeps main pipeline moving	The failure detail (exception message, stack) isn't queryable structured data by default
2. Log & skip + save to DB	Logs the failure and writes a structured failure record into a database table	Queryable, alertable failure records for ops dashboards	No automatic republish path back into the main flow
3. Hybrid	Both: publish to DLT and persist a failure record	Best of both — replayable from the DLT, queryable from the DB	More moving parts to keep consistent

@Bean
public DeadLetterPublishingRecoverer recoverer(KafkaTemplate<Object,Object> template) {
    return new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
}

public class HybridRecoverer implements ConsumerRecordRecoverer {

    private final DeadLetterPublishingRecoverer dltRecoverer;
    private final FailureRecordRepository failureRepository;

    @Override
    public void accept(ConsumerRecord<?,?> record, Exception ex) {
        dltRecoverer.accept(record, ex);
        failureRepository.save(new FailureRecord(
            record.topic(), record.partition(), record.offset(),
            (String) record.value(), ex.getMessage()));
    }
}

▲ Pitfall

A Dead Letter Topic without a defined reprocessing plan is just a graveyard — records accumulate there and nobody looks at them until an incident forces someone to. Treat "who monitors the DLT, and how do records get replayed" as a required part of the design, not an afterthought.

✓ Quick recap

What happens to a partition by default (zero custom config) when a listener throws? Spring Boot's auto-configured DefaultErrorHandler retries the record a bounded number of times (~10 attempts), then logs and lets the offset commit anyway — the partition is NOT stuck, but the failed record is silently lost unless you add a recoverer. What decides whether DefaultErrorHandler retries at all, versus going straight to the recoverer? Whether the exception type is classified retryable or not. What's the risk of a Dead-Letter-only recovery strategy with no monitoring? Failed records silently accumulate with no one aware or acting on them.

💻 Code example

@Bean
public DeadLetterPublishingRecoverer recoverer(KafkaTemplate<Object,Object> template) {
    return new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
}

Want a visual for this concept?

Generate a diagram tailored to “Consumer Errors, Retry & Recovery” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Consumer Timing Configurations →← Back to all Kafka & Microservices chapters