5.2 Docker Environment

Setting up your local docker environment for this lab.

Changes to the docker-compose file

The setup for this lab is similar as the previous. However, we use Kafka Connect with Debezium PostgreSQL connector to write messages to Kafka.

Environment

Task 5.2.1 - Setting up Kafka Connect with Debezium

For Kafka Connect we will use a pre-built docker container which is based on the Kafka Strimzi image.

  • Add the following container configuration to the docker/docker-compose.yaml file:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
  connect:
    image: quay.io/puzzle/microservices-lab-debezium-connect:latest
    hostname: connect
    container_name: debezium-connect
    ports:
     - 8083:8083
    networks:
      - debezium
    depends_on:
     - kafka
     - order-db
     - stock-db
     - zookeeper
    command: /opt/kafka/kafka_connect_run.sh
    environment:
     - KAFKA_CONNECT_BOOTSTRAP_SERVERS=kafka:9092
     - |
         KAFKA_CONNECT_CONFIGURATION=
         key.converter=org.apache.kafka.connect.json.JsonConverter
         value.converter=org.apache.kafka.connect.json.JsonConverter
         key.converter.schemas.enable=false
         value.converter.schemas.enable=false
         group.id=connect-debezium
         offset.storage.topic=connect-debezium-offsets
         config.storage.topic=connect-debezium-configs
         status.storage.topic=connect-debezium-status
         offset.storage.replication.factor=1
         config.storage.replication.factor=1
         status.storage.replication.factor=1
         consumer.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor
         producer.interceptor.classes=io.opentracing.contrib.kafka.TracingProducerInterceptor         
     - KAFKA_CONNECT_METRICS_ENABLED=true
     - JAEGER_SERVICE_NAME=debezium-connect
     - JAEGER_AGENT_HOST=jaeger
     - STRIMZI_TRACING=jaeger

Task 5.2.2 - Changing database docker image

The Debezium PostgreSQL Connector needs access to the WAL log. We therefore switch to the quay.io/debezium/postgres:13 image maintained by Debezium. This image provides us the configurations and permissions needed to access the WAL log.

Open docker/docker-compose.yaml and make the following changes:

  • order-db: change image to quay.io/debezium/postgres:13
  • stock-db: change image to quay.io/debezium/postgres:13
Task Hint

Except the docker image for the order microservice your docker compose should look like this one:

full source docker-compose.yaml