advanced~5h

Microservices Persistence — Database-per-Service, Outbox Pattern & CDC

Database-per-service breaks every cross-entity transactional guarantee this course has relied on so far — the outbox pattern and CDC are the specific tools that restore reliable consistency across independently-owned service databases.

Learning objectives

Explain the dual-write problem and why a naive database-update-then-network-call approach is unsafe.
Implement the outbox pattern to atomically persist a business change and its corresponding event.
Explain what CDC/Debezium does differently from application-level event publishing, and why consumers must be idempotent.

Everything in this course so far assumed one application, one database, one transaction boundary. Microservices break that assumption entirely.

📖 Story

Imagine a wedding where the caterer and florist each have their OWN separate booking system, by design. The planner needs to both confirm catering AND notify the florist — but can't wrap both in one atomic transaction, since they're separate systems. Here's the exact bug this creates in code:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);              // saved to OUR database — committed
    inventoryServiceClient.decrementStock(order);   // ⚠️ separate network call!
    // If this call fails (a crash, a network blip) AFTER our save already
    // committed, the order exists but inventory was never actually decremented.
    // The two systems have silently diverged. This is the dual-write problem.
}

Here's the fix — the Outbox Pattern:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);
    outboxRepository.save(new OutboxEvent("OrderCreated", order.getId(), toJson(order)));
    // BOTH inserts are in the SAME local transaction — genuinely atomic,
    // exactly like any other multi-table write from earlier in this course.
}

A separate, independent process then reads unpublished outbox rows and reliably publishes them to Kafka, retrying until successful — closing the gap the direct network call left open.

Database-per-service — each service owns its own database exclusively. Outbox pattern — writing a "message to publish" row into the SAME local transaction as your business data change. CDC (Change Data Capture) — capturing row-level changes from a database's transaction log. Debezium — a popular CDC tool, commonly publishing to Kafka.

Let's finish this chapter's outbox example with the publishing side.

@Scheduled(fixedDelay = 1000)
public void publishPendingEvents() {
    List<OutboxEvent> pending = outboxRepository.findByPublishedFalse();
    for (OutboxEvent event : pending) {
        kafkaTemplate.send("orders-topic", event.getPayload());
        event.setPublished(true);
        outboxRepository.save(event);
    }
}

This polling approach works, but Debezium offers something more elegant: instead of your application code explicitly writing to (and polling) an outbox table, Debezium reads the database's own transaction log directly — the same low-level mechanism the database uses for its own replication — and publishes every committed row change automatically, with ZERO application code needed to explicitly "publish" anything.

Consumers must be idempotent

@KafkaListener(topics = "orders-topic")
public void handleOrderCreated(OrderCreatedEvent event) {
    if (processedEventRepository.existsById(event.getId())) {
        return;   // already handled — outbox/CDC delivery is AT-LEAST-ONCE, not exactly-once
    }
    // ... actual processing ...
    processedEventRepository.save(new ProcessedEvent(event.getId()));
}

The outbox pattern's atomicity comes from inserting into your business table AND the outbox table being two INSERT statements in the SAME database transaction — exactly Chapter 11's transaction atomicity, just applied to a message-to-be-published row. Debezium connects to the database as a logical replication client (PostgreSQL's logical decoding, or MySQL's binlog) — it doesn't poll or query your tables at all; it reads the same change stream the database uses internally.

This chapter's exact order-creation example — publishing an "OrderCreated" event that inventory and notification services both consume — is the standard, textbook real-world outbox use case.
A CDC pipeline streaming changes from a legacy monolith's database into a new microservice's database is a common real migration strategy.

Assume database-per-service means NO cross-service ACID transactions are possible, ever — design your consistency strategy deliberately, not after a dual-write bug in production, exactly this chapter's opening story.
Prefer the outbox pattern over a naive database-then-network-call approach whenever a business event must be reliably published.
Design consumers to be idempotent, like this chapter's processedEventRepository check — outbox/CDC delivery is at-least-once, not exactly-once.

⚠️ Why this keeps happening

The dual-write problem is invisible in almost every test — a database update followed by a network call succeeds nearly every time in a healthy test environment; the failure mode (a crash landing in the narrow window between the two operations) only shows up under real production instability.

Doing a database update followed by a separate, non-transactional network call, exactly this chapter's opening bug — discovering the dual-write problem only during a real incident.
Building consumers that assume exactly-once delivery, when the actual guarantee is at-least-once — a non-idempotent consumer processes a duplicate as if new.
Forgetting to prune the outbox table over time, letting it grow unboundedly.

The outbox pattern adds one extra INSERT per business transaction — small, generally negligible compared to the correctness guarantee it buys. CDC/Debezium has essentially no impact on the source database's normal query performance, since it never polls or queries tables at all.

Outbox/CDC pipelines streaming data to other services need the same access-control rigor as any cross-service data flow — an outbox table exposing more fields than a consumer actually needs is an easy-to-overlook data-exposure risk.

Monitor outbox table growth and publishing lag as a first-class metric — a growing backlog signals either a stalled publishing process or a struggling downstream broker.

Build the outbox-publishing/CDC pipeline with the same monitoring rigor as any other critical production infrastructure — it's the single mechanism keeping otherwise-independent services eventually consistent.

Reproduce this chapter's exact dual-write bug (database save, then a simulated failing network call) and observe the resulting inconsistency.
Implement the outbox pattern instead — one local transaction for both the business entity and the outbox row — and build a simple polling publisher.
Simulate a consumer receiving the same event twice, and confirm your idempotency check (this chapter's processedEventRepository pattern) prevents a duplicate side effect.

✓ Quick recap

Database-per-service means no cross-service ACID transactions exist — a naive database-then-network-call approach has the unavoidable dual-write problem, exactly this chapter's opening story.
The outbox pattern writes an event into the SAME local transaction as your business data change, giving genuine atomicity.
CDC/Debezium reads a database's transaction log directly; consumers must be idempotent since delivery is at-least-once.

Want a visual for this concept?

Generate a diagram tailored to “Microservices Persistence — Database-per-Service, Outbox Pattern & CDC” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to Testing the Persistence Layer — @DataJpaTest & Testcontainers →← Back to all Spring Data JPA & Hibernate Mastery chapters