Debezium Connector 실습 환경 구축하기

 

본 글에서는 Debezium Connector 의 실습 환경을 세팅하기 위해서 docker-compose.yaml 을 세팅합니다.

3개의 Kafka Broker 와 3개의 Kafka Connect Worker 들로 구성된 클러스터를 구축합니다.

그리고 kafka-ui 와 kafka-connect-ui 라는 Kafka 와 관련된 관리자 Web UI 를 함께 실행합니다.

 

참고로 Kafka Connector Worker 의 경우에는 Debezium Connector Plugin 을 추가한 Docker Image 를 사용하였습니다.

 

docker network create kafka

cat <<'EOF'> /tmp/docker-compose.yaml
version: '3.8'

services:
  kafka1:
    image: confluentinc/cp-kafka:7.8.0
    hostname: kafka1
    container_name: kafka1
    networks:
      - kafka
    environment:
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: 'INTERNAL://:9092,CONTROLLER://:9093'
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
    restart: always

  kafka2:
    image: confluentinc/cp-kafka:7.8.0
    hostname: kafka2
    container_name: kafka2
    networks:
      - kafka
    environment:
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: 'INTERNAL://:9092,CONTROLLER://:9093'
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
    restart: always

  kafka3:
    image: confluentinc/cp-kafka:7.8.0
    hostname: kafka3
    container_name: kafka3
    networks:
      - kafka
    environment:
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: 'INTERNAL://:9092,CONTROLLER://:9093'
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
    restart: always

  kafka-ui:
    image: provectuslabs/kafka-ui:v0.7.2
    container_name: kafka-ui
    networks:
      - kafka
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
      KAFKA_CLUSTERS_0_NAME: cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka1:9092,kafka2:9092,kafka3:9092
    ports:
      - 8080:8080
    restart: always

  connect1:
    image: westlife0615/debezium-connector:1
    hostname: connect1
    container_name: connect1
    networks:
      - kafka
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092
      CONNECT_REST_ADVERTISED_HOST_NAME: connect1
      CONNECT_GROUP_ID: connect-cluster
      CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-status
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
    restart: always
  connect2:
    image: westlife0615/debezium-connector:1
    hostname: connect2
    container_name: connect2
    networks:
      - kafka
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092
      CONNECT_REST_ADVERTISED_HOST_NAME: connect2
      CONNECT_GROUP_ID: connect-cluster
      CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-status
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
    restart: always
  connect3:
    image: westlife0615/debezium-connector:1
    hostname: connect3
    container_name: connect3
    networks:
      - kafka
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092
      CONNECT_REST_ADVERTISED_HOST_NAME: connect3
      CONNECT_GROUP_ID: connect-cluster
      CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-status
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
    restart: always
  kafka-connect-ui:
    image: landoop/kafka-connect-ui
    container_name: kafka-connect-ui
    hostname: kafka-connect-ui
    networks:
      - kafka
    ports:
      - "8000:8000"
    environment:
      - CONNECT_URL=http://connect1:8083,http://connect2:8083,http://connect3:8083
    restart: always


networks:
  kafka:
    driver: bridge
    external: true
EOF


docker-compose -f /tmp/docker-compose.yaml -p kafka-connect up -d

 

위 docker-compose.yaml 을 실행하게 되면 아래와 같은 Docker Container 결과를 확인할 수 있습니다.

 

 

그리고 8080 Port 를 통해서 Kafka UI Web Interface 에 접속할 수 있습니다.

아래와 같은 페이지에서 Kafka Admin 과 관련된 기능들을 수행하실 수 있습니다.

 

 

 

8000 Port 를 통해서 Kafka Connect UI 에 접속할 수 있습니다.

접속한 화면의 예시는 아래와 같습니다.

 

 

 

Debezium Plugin 을 추가한 Kafka Connect Worker Image.

 

https://hub.docker.com/r/westlife0615/debezium-connector

https://hub.docker.com/r/westlife0615/debezium-connector