advanced~2h

Transactions & Exactly-Once Semantics

"Exactly once" is the most misunderstood phrase in Kafka. This module defines it precisely, then builds it.

Learning objectives

Beginner: State precisely what Kafka's "exactly-once" guarantee actually covers, and the common misconception about what it doesn't.
Intermediate: Use @Transactional or KafkaTemplate.executeInTransaction() to publish a message as part of a transactional unit of work.
Advanced: Publish multiple messages atomically and reason about exactly-once semantics on the consumer side of a read-process-write pipeline.

◆ The problem

Every mechanism covered so far — idempotent producer, manual acknowledgment, upsert-safe persistence — solves one piece of duplicate-safety. None of them alone guarantees that a batch of multiple related writes either all happen or none do. If publishing event A succeeds but event B (part of the same logical operation) fails, you're left in a half-done state no single-record mechanism can fix.

Exactly-once semantics (EOS) in Kafka specifically means: a set of reads-and-writes performed as one transaction either all become visible to downstream consumers, or none do — atomicity across multiple sends (and, on the consumer side, across a read-process-write cycle). It is built from idempotence (no duplicate writes on retry) plus transactions (atomicity across multiple writes) — neither piece alone is "exactly once."

◆ Under the hood

A Kafka producer transaction works by tagging all writes within it with a transactional ID and committing a special marker record once every write in the transaction has succeeded. Consumers configured with isolation.level=read_committed (the safe default to pair with transactional producers) simply never surface records from a transaction until that commit marker is written — an aborted transaction's records are filtered out entirely, invisible to well-behaved consumers, rather than needing to be manually rolled back.

@Bean
public KafkaTransactionManager<String,String> kafkaTransactionManager(ProducerFactory<String,String> pf) {
    return new KafkaTransactionManager<>(pf);
}

@Service
public class LibraryEventProducer {

    @Transactional("kafkaTransactionManager")
    public void publishAsTransaction(LibraryEvent event) throws JsonProcessingException {
        kafkaTemplate.send("library-events", objectMapper.writeValueAsString(event));
        kafkaTemplate.send("library-events-audit", objectMapper.writeValueAsString(event));
        // both sends commit together, or neither is visible to consumers
    }
}

spring:
  kafka:
    producer:
      transaction-id-prefix: library-events-tx-

💻 Code example

@Bean
public KafkaTransactionManager<String,String> kafkaTransactionManager(ProducerFactory<String,String> pf) {
    return new KafkaTransactionManager<>(pf);
}

@Service
public class LibraryEventProducer {

    @Transactional("kafkaTransactionManager")
    public void publishAsTransaction(LibraryEvent event) throws JsonProcessingException {
        kafkaTemplate.send("library-events", objectMapper.writeValueAsString(event));
        kafkaTemplate.send("library-events-audit", objectMapper.writeValueAsString(event));
        // both sends commit together, or neither is visible to consumers
    }
}

For imperative, programmatic control (rather than declarative AOP-based @Transactional):

kafkaTemplate.executeInTransaction(ops -> {
    ops.send("library-events", eventJson);
    ops.send("library-events-audit", eventJson);
    return true;
});

	@Transactional	executeInTransaction()
Style	Declarative — AOP proxy manages the boundary	Imperative — you define the boundary explicitly in code
Good for	Service-layer methods that are naturally "one transactional unit"	Fine-grained or conditional transactional logic inside a larger method

💻 Code example

kafkaTemplate.executeInTransaction(ops -> {
    ops.send("library-events", eventJson);
    ops.send("library-events-audit", eventJson);
    return true;
});

Both approaches above already demonstrate this — the defining feature of a Kafka transaction is exactly that it can span multiple send() calls, potentially to different topics, and they become visible to read_committed consumers as one atomic unit.

The most valuable pattern combines both sides: a consumer reads a record, does some processing, and writes both a database update and a new outgoing Kafka record — the classic "read-process-write" loop.

▲ Common mistake

Just putting @Transactional("kafkaTransactionManager") on a listener that does BOTH a JPA save and a kafkaTemplate.send(), expecting them to commit atomically together. KafkaTransactionManager only manages the Kafka producer transaction — it has no idea your JPA EntityManager exists. A crash between the two lines below would NOT roll back the DB write; only the Kafka send is protected.

@Transactional("kafkaTransactionManager")   // ⚠️ this alone does NOT cover the DB write
@KafkaListener(topics = "library-events")
public void onMessage(ConsumerRecord<String,String> record) {
    libraryEventRepository.save(toEntity(parse(record.value())));   // NOT covered by this transaction manager
    kafkaTemplate.send("library-events-processed", record.value()); // this part IS covered
}

To genuinely make the DB write and the Kafka send commit or roll back together, chain the two transaction managers with ChainedKafkaTransactionManager:

@Bean
public ChainedKafkaTransactionManager<Object,Object> chainedTransactionManager(
        KafkaTransactionManager<Object,Object> kafkaTm, JpaTransactionManager jpaTm) {
    return new ChainedKafkaTransactionManager<>(kafkaTm, jpaTm);
}

@Transactional("chainedTransactionManager")
@KafkaListener(topics = "library-events")
public void onMessage(ConsumerRecord<String,String> record) {
    libraryEventRepository.save(toEntity(parse(record.value())));   // now genuinely covered
    kafkaTemplate.send("library-events-processed", record.value());
    // commits together — a crash between these two lines rolls both back on restart
}

For the narrower "consume, transform, produce" case where you only care about Kafka-to-Kafka atomicity (no DB involved), Spring Kafka has a more targeted tool: kafkaTemplate.executeInTransaction() combined with sendOffsetsToTransaction(), which writes the consumer's offset commit itself as PART of the same Kafka transaction as the outgoing send — instead of committing the offset the normal way, it's folded into the transaction, so the offset only advances if the downstream publish actually succeeded.

▲ Pitfall

Transactions guarantee atomicity of Kafka writes (and, with ChainedKafkaTransactionManager, a genuinely chained DB write) — they do not make an arbitrary external side effect (e.g. calling a third-party HTTP API inside the listener) transactional. If your listener calls out to a non-transactional system, that call can still happen more than once on retry even inside a "transactional" listener — scope transactions to what's actually transactional.

✓ Quick recap

Is idempotence alone "exactly-once semantics"? No — idempotence prevents duplicate writes from retries of a single record; EOS additionally requires atomicity across multiple related writes, which needs transactions. Does plain @Transactional("kafkaTransactionManager") cover a JPA save alongside a Kafka send? No — KafkaTransactionManager only manages the Kafka side; a co-located DB write needs ChainedKafkaTransactionManager to actually be chained in. How do read_committed consumers handle an aborted transaction's records? They never see them at all — the records are filtered out, not something the consumer has to detect and roll back itself. Do Kafka transactions make a call to an external, non-Kafka, non-transactional system safe from duplication? No — transaction atomicity covers Kafka (and a properly chained DB), not arbitrary external side effects.

💻 Code example

@Bean
public ChainedKafkaTransactionManager<Object,Object> chainedTransactionManager(
        KafkaTransactionManager<Object,Object> kafkaTm, JpaTransactionManager jpaTm) {
    return new ChainedKafkaTransactionManager<>(kafkaTm, jpaTm);
}

@Transactional("chainedTransactionManager")
@KafkaListener(topics = "library-events")
public void onMessage(ConsumerRecord<String,String> record) {
    libraryEventRepository.save(toEntity(parse(record.value())));   // now genuinely covered
    kafkaTemplate.send("library-events-processed", record.value());
    // commits together — a crash between these two lines rolls both back on restart
}

Want a visual for this concept?

Generate a diagram tailored to “Transactions & Exactly-Once Semantics” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Health, Monitoring & Observability →← Back to all Kafka & Microservices chapters