advanced~2h

Kafka Transactions & Exactly-Once Semantics

Chapter 17 kept saying "idempotency is required because delivery is at-least-once." This chapter is the mechanism Kafka itself provides to tighten that guarantee — and exactly what "exactly-once" does and doesn't promise.

Guarantee	What can go wrong
At-most-once	A message can be lost (e.g. commit the consumer offset before processing — a crash after the offset commit but before processing means the message is skipped forever).
At-least-once (Kafka's default, and what Chapter 17 assumed)	A message can be redelivered and processed more than once, but is never silently lost.
Exactly-once (EOS)	Neither lost nor duplicated — built from two mechanisms working together (§19.2, §19.3), not a single setting.

◆ The problem

A producer sends a message; the network glitches before the broker's acknowledgment arrives back; the producer, not knowing whether the send actually succeeded, retries. If the original send did succeed, the broker now has the same message written twice.

spring.kafka.producer.properties.enable.idempotence=true

◆ Under the hood

An idempotent producer is assigned a unique Producer ID by the broker and tags every message with a monotonically increasing sequence number per partition. The broker tracks the last sequence number it accepted from each producer/partition pair and simply discards (rather than re-appends) any retry carrying a sequence number it's already seen — turning "retry might duplicate" into "retry is always safe," at the producer-to-broker level specifically.

💻 Code example

spring.kafka.producer.properties.enable.idempotence=true

◆ The problem

The idempotent producer alone only protects a single send from duplication. A common real pattern — "read a message from topic A, process it, write a result to topic B, and commit the consumer's offset on topic A" — involves multiple distinct operations (a produce to B, an offset commit on A) that need to succeed or fail together, the exact same all-or-nothing shape from Chapter 01, now applied inside Kafka itself.

spring.kafka.producer.transaction-id-prefix=order-processor-tx-

@Transactional("kafkaTransactionManager")
@KafkaListener(topics = "orders-raw")
public void processOrder(ConsumerRecord<String,String> record) {
    EnrichedOrder enriched = enrichOrder(record.value());
    kafkaTemplate.send("orders-enriched", enriched.toJson());
    // the produce to "orders-enriched" AND the consumer offset commit on "orders-raw"
    // are committed together as ONE Kafka transaction — both happen, or neither does
}

◆ Under the hood — the transaction marker

Kafka transactions work by writing a special commit marker record to every partition involved, only after every write in the transaction has succeeded — this is directly analogous to Chapter 02's undo-log-based atomicity, just implemented as an append-only marker in the log itself rather than a separate undo structure. Consumers configured for read_committed (§19.4) simply skip over records belonging to a transaction that never got its commit marker written (an aborted transaction) — they never see them at all, rather than needing to detect and roll back an already-read message themselves.

💻 Code example

@Transactional("kafkaTransactionManager")
@KafkaListener(topics = "orders-raw")
public void processOrder(ConsumerRecord<String,String> record) {
    EnrichedOrder enriched = enrichOrder(record.value());
    kafkaTemplate.send("orders-enriched", enriched.toJson());
    // the produce to "orders-enriched" AND the consumer offset commit on "orders-raw"
    // are committed together as ONE Kafka transaction — both happen, or neither does
}

spring.kafka.consumer.properties.isolation.level=read_committed

Isolation level	Behavior
read_uncommitted (default)	Consumers see every message, including ones from a transaction that later aborts — directly analogous to SQL's dirty read (Chapter 04).
read_committed	Consumers only ever see messages from transactions that actually committed — aborted transactions' messages are filtered out entirely.

▲ Common mistake

Enabling transactional producers without also setting the consumer's isolation.level=read_committed gives you none of the actual benefit — the consumer defaults to read_uncommitted and will happily read messages from transactions that later abort, exactly the dirty-read problem transactions were supposed to prevent. Both sides must be configured correctly together.

💻 Code example

spring.kafka.consumer.properties.isolation.level=read_committed

spring:
  kafka:
    producer:
      properties:
        enable.idempotence: true
      transaction-id-prefix: order-processor-tx-
    consumer:
      properties:
        isolation.level: read_committed
      enable-auto-commit: false # offset commits happen as part of the Kafka transaction instead

Scope	Covered by Kafka EOS?
Kafka-to-Kafka: consume from A, produce to B, atomically	Yes — this is exactly what Kafka transactions guarantee.
Kafka-to-external-database write, atomically	No — writing to Postgres and Kafka together is the exact same dual-write problem from Chapter 15 §3; Kafka's transactions don't extend to external systems. Use the Outbox Pattern (Chapter 18) for that case instead.

💻 Code example

spring:
  kafka:
    producer:
      properties:
        enable.idempotence: true
      transaction-id-prefix: order-processor-tx-
    consumer:
      properties:
        isolation.level: read_committed
      enable-auto-commit: false # offset commits happen as part of the Kafka transaction instead

Want a visual for this concept?

Generate a diagram tailored to “Kafka Transactions & Exactly-Once Semantics” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to System Design with Transactions →← Back to all Transaction Mastery chapters