5. Debezium with Outbox Pattern

Introduction

In this lab we will use a different integration pattern. The previous lab using Kafka messaging has a certain design flaw. The microservices are writing to the database using a local transaction. however, this transaction does not span the writing to the Kafka topic. if a rollback occurs, only the data written to the database is rolled back. Handling such cases would need to be handled explicitly in the application code.

In this lab we will use the Outbox Pattern1 to get rid of this flaw.

For this to achieve we will use Kafka Connect with a Debezium PostgreSQL source connector.

Kafka Connect and Debezium

Kafka Connect also provides the functionality to use Single Message Transformations (SMT) to modify a message before it is sent to the Kafka Topic. The Debezium connector also includes an Outbox Event Router2.

Transactional Outbox Pattern

Databases usually write their changes to a write-ahead logfile (WAL) before actually writing them to the database. This provides atomicity and durability.

Debezium makes use of this by reading the WAL to detect changes in the database. This technique is called Change-Data-Capture (CDC)3 and is a widely used pattern to integrate legacy systems and applications.

The outbox pattern specifies that beside our business data there will also be an event-record which is written to the dedicated outboxevent table. As that will take place in the same local transaction, we have strong guarantee that our data is either written or rolled-back completely including the event.

In Kafka Connect our Debezium connector will monitor the WAL log and watch out for INSERT to the outboxevent table. Finding a new record in the WAL log Debezium will generate a Kafka Message to a designated topic. Debezium provides an EventRouter which is implemented as Single Message Transformation. This EventRouter will form our actual Kafka Message using the information from the outboxevent table (actually the WAL log).

Our application will therefore only consume messages from Kafka topics.

Environment

This integration pattern is highly decoupled. Either the Kafka consumers in our application as debezium are able to recover from where they left of. This guarantees that our application is able to process its tasks if the database is available. Any downtime or network incident between Debezium (Kafka Connect) or Kafka itself will only result in delayed processing of the events.

Lab Overview

At the end of this lab our environment is expected to look like this. In comparison to the last lab we will introduce the Kafka connect service with our debezium PostgreSQL connector. Further we will create additional outboxevent tables and our applications will stop to write to Kafka directly. We will still use saga choreography as in the previous lab.

Environment


  1. Transactional Outbox Pattern: https://microservices.io/patterns/data/transactional-outbox.html ↩︎

  2. Outbox Event Router: https://debezium.io/documentation/reference/configuration/outbox-event-router.htm ↩︎

  3. Change Data Capture: https://en.wikipedia.org/wiki/Change_data_capture ↩︎


5.1 Getting Started

Getting started with the lab environment.

5.2 Docker Environment

Setting up your local docker environment for this lab.

5.3 Microservices Setup

Understand the setup of the Order microservice.

5.4 Preparing the Database

Setup the outbox and event history table for use with debezium.

5.5 Define Events

Define the events used.

5.6 Event Handling

Fire and consume events.

5.7 Docker build

Building the docker image for the order microservice.

5.8 Running the Lab

Run the Lab with Docker Compose