[Clickhouse] Shard & Replica Cluster 구성하기

Database/Clickhouse 2024. 2. 14. 07:14

728x90

- 목차

들어가며.

이번 글에서는 docker-compose 를 활용하여 Clickhouse Cluster 를 구성하는 내용을 작성하려고 합니다.

ClickHouse 의 docker-compose Recipes 에서 설명하는 내용을 토대로 작성하였구요.

Shard 와 Replica 의 다양한 조합과 Clickhouse-Keeper 가 함께 클러스터를 구성합니다.

아래 링크는 Clickhouse docker-compose Recipes 에 대한 Github 링크입니다.

https://github.com/ClickHouse/examples/tree/main/docker-compose-recipes/recipes

https://github.com/ClickHouse

ClickHouse

ClickHouse has 177 repositories available. Follow their code on GitHub.

github.com

1 Shard & 1 Clickhouse-Keeper.

하나의 Shard 와 1개의 Zookeeper 를 구성하는 Clickhouse Cluster 를 구성합니다.

ch-1S_1K Clickhouse Cluster 실행하기.

먼저 아래 명령어들을 실행하여 Clickhouse Cluster 를 실행합니다.

아래 명령어들은 Clickhouse 의 config.xml, user.xml 그리고 Zookeeper 의 config.xml 파일을 생성하는 명령어와

docker-compose yaml 파일을 생성하는 명령어들입니다.

< Clickhouse Config xml >

cat <<EOF> /tmp/clickhouse_config.xml
<clickhouse replace="true">
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <display_name>ch-1S_1K</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <user_directories>
        <users_xml>
            <path>users.xml</path>
        </users_xml>
        <local_directory>
            <path>/var/lib/clickhouse/access/</path>
        </local_directory>
    </user_directories>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <zookeeper>
        <node>
            <host>clickhouse-keeper</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>
EOF

< Clickhouse Keeper Config xml >

cat <<EOF> /tmp/keeper_config.xml
<clickhouse replace="true">
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>information</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
EOF

< docker compose yaml >

cat <<EOF> /tmp/ch-1S_1K.yaml
version: '3.8'
services:
  clickhouse:
    image: 'clickhouse/clickhouse-server:23.4'
    user: '101:101'
    container_name: clickhouse
    hostname: clickhouse
    volumes:
      - /tmp/clickhouse_config.xml:/etc/clickhouse-server/config.d/config.xml
    ports:
      - '127.0.0.1:8123:8123'
      - '127.0.0.1:9000:9000'
    depends_on:
      - clickhouse-keeper
  clickhouse-keeper:
    image: 'clickhouse/clickhouse-keeper:23.4-alpine'
    user: '101:101'
    container_name: clickhouse-keeper
    hostname: clickhouse-keeper
    volumes:
      - /tmp/keeper_config.xml:/etc/clickhouse-keeper/keeper_config.xml
    ports:
      - '127.0.0.1:9181:9181'
EOF

< docker-compose up >

docker-compose -f /tmp/ch-1S_1K.yaml -p ch-cluster up -d

위 단계를 마치게 되면 아래와 같이 2개의 Docker Container 가 구성됩니다.

하나의 Clickhouse Container 와 Clickhouse-Keeper Container 가 실행됩니다.

MergeTree Table 생성하기.

Replica 구성을 하지 않아서 ReplicatedMergeTree 와 같은 테이블 엔진은 사용할 수 없습니다.

대신 MergeTree Table 을 생성해보도록 하겠습니다.

< Clickhouse Client Shell 활성화 >

docker exec -it clickhouse clickhouse-client

< MergeTree Table 생성 >

create table default.user_actions (
    user String,
    action String,
    acted_at DateTime
) engine=MergeTree()
order by acted_at
partition by toYYYYMM(acted_at);

< 데이터 생성 >

insert into default.user_actions(user, action, acted_at)
values ('Andy', 'Visit', '2023-01-01 00:00:00'),
('Brian', 'Buy', '2023-01-01 00:01:00'),
('Dennis', 'View', '2023-01-01 00:02:00'),
('Chris', 'View', '2023-01-01 00:03:00'),
('Emma', 'Buy', '2023-01-01 00:04:00'),
('Fabian', 'Cart', '2023-01-01 00:05:00'),
('Gareth', 'RemoveCart', '2023-01-01 00:06:00'),
('Hayden', 'Cart', '2023-01-01 00:07:00'),
('Illa', 'View', '2023-01-01 00:08:00')

Query id: 062230b2-8ae8-4ff2-ac97-b5f44f37dfae

Ok.

9 rows in set. Elapsed: 0.002 sec.

< 테이블 데이터 조회 >

select * from user_actions

Query id: 2bebd066-7976-47ea-885e-1b341405837e

┌─user───┬─action─────┬────────────acted_at─┐
│ Andy   │ Visit      │ 2023-01-01 00:00:00 │
│ Brian  │ Buy        │ 2023-01-01 00:01:00 │
│ Dennis │ View       │ 2023-01-01 00:02:00 │
│ Chris  │ View       │ 2023-01-01 00:03:00 │
│ Emma   │ Buy        │ 2023-01-01 00:04:00 │
│ Fabian │ Cart       │ 2023-01-01 00:05:00 │
│ Gareth │ RemoveCart │ 2023-01-01 00:06:00 │
│ Hayden │ Cart       │ 2023-01-01 00:07:00 │
│ Illa   │ View       │ 2023-01-01 00:08:00 │
└────────┴────────────┴─────────────────────┘

9 rows in set. Elapsed: 0.004 sec.

1 Shard & 2 Replicas 구성하기.

이번에는 1개의 Shard 와 2개의 Replica 로 구성된 Clickhouse Cluster 를 구성해보겠습니다.

이번 구성을 위해서는 2개의 Clickhouse Config xml 과 3개의 Clickhouse Keeper xml 이 필요합니다.

생성해야하는 파일의 양이 많은 관계로 아래에 Toggle 형식으로 파일 내용을 첨부하였습니다.

< clickhouse-01 config.xml >

cat <<EOF> /tmp/clickhouse_01_config.xml
<clickhouse replace="true">
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <display_name>cluster_1S_2R node 1</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <user_directories>
        <users_xml>
            <path>users.xml</path>
        </users_xml>
        <local_directory>
            <path>/var/lib/clickhouse/access/</path>
        </local_directory>
    </user_directories>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <remote_servers >
        <cluster_1S_2R>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-01</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-02</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster_1S_2R>
    </remote_servers>
    <zookeeper>
        <node>
            <host>clickhouse-keeper-01</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-02</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-03</host>
            <port>9181</port>
        </node>
    </zookeeper>
    <macros>
        <shard>01</shard>
        <replica>01</replica>
        <cluster>cluster_1S_2R</cluster>
    </macros>
</clickhouse>
EOF

< clickhouse-02 config.xml >

cat <<EOF> /tmp/clickhouse_02_config.xml
<clickhouse replace="true">
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <display_name>cluster_1S_2R node 2</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <user_directories>
        <users_xml>
            <path>users.xml</path>
        </users_xml>
        <local_directory>
            <path>/var/lib/clickhouse/access/</path>
        </local_directory>
    </user_directories>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <remote_servers>
        <cluster_1S_2R>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-01</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-02</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster_1S_2R>
    </remote_servers>
    <zookeeper>
        <node>
            <host>clickhouse-keeper-01</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-02</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-03</host>
            <port>9181</port>
        </node>
    </zookeeper>
    <macros>
        <shard>01</shard>
        <replica>02</replica>
        <cluster>cluster_1S_2R</cluster>
    </macros>
</clickhouse>
EOF

< keeper_config 1 xml >

cat <<EOF> /tmp/keeper_01_config.xml
<clickhouse replace="true">
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>information</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
EOF

< keeper_config 2 xml >

cat <<EOF> /tmp/keeper_02_config.xml
<clickhouse replace="true">
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>2</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>information</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
EOF

< keeper_config 3 xml >

cat <<EOF> /tmp/keeper_03_config.xml
<clickhouse replace="true">
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>3</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>information</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
EOF

cat <<EOF> /tmp/clickhouse.yaml
version: '3.8'
services:
  clickhouse-01:
    image: "clickhouse/clickhouse-server:23.4"
    user: "101:101"
    container_name: clickhouse-01
    hostname: clickhouse-01
    volumes:
      - /tmp/clickhouse_01_config.xml:/etc/clickhouse-server/config.d/config.xml
    ports:
      - "127.0.0.1:8123:8123"
      - "127.0.0.1:9000:9000"
    depends_on:
      - clickhouse-keeper-01
      - clickhouse-keeper-02
      - clickhouse-keeper-03
  clickhouse-02:
    image: "clickhouse/clickhouse-server:23.4"
    user: "101:101"
    container_name: clickhouse-02
    hostname: clickhouse-02
    volumes:
      - /tmp/clickhouse_02_config.xml:/etc/clickhouse-server/config.d/config.xml
    ports:
      - "127.0.0.1:8124:8123"
      - "127.0.0.1:9001:9000"
    depends_on:
      - clickhouse-keeper-01
      - clickhouse-keeper-02
      - clickhouse-keeper-03
  clickhouse-keeper-01:
    image: "clickhouse/clickhouse-keeper:23.4-alpine"
    user: "101:101"
    container_name: clickhouse-keeper-01
    hostname: clickhouse-keeper-01
    volumes:
     - /tmp/keeper_01_config.xml:/etc/clickhouse-keeper/keeper_config.xml
    ports:
        - "127.0.0.1:9181:9181"
  clickhouse-keeper-02:
    image: "clickhouse/clickhouse-keeper:23.4-alpine"
    user: "101:101"
    container_name: clickhouse-keeper-02
    hostname: clickhouse-keeper-02
    volumes:
     - /tmp/keeper_02_config.xml:/etc/clickhouse-keeper/keeper_config.xml
    ports:
        - "127.0.0.1:9182:9181"
  clickhouse-keeper-03:
    image: "clickhouse/clickhouse-keeper:23.4-alpine"
    user: "101:101"
    container_name: clickhouse-keeper-03
    hostname: clickhouse-keeper-03
    volumes:
     - /tmp/keeper_03_config.xml:/etc/clickhouse-keeper/keeper_config.xml
    ports:
        - "127.0.0.1:9183:9181"
EOF

docker-compose -f /tmp/clickhouse.yaml -p clickhouse-cluster up -d

위 명령어들을 수행하게 되면 아래와 같이 2개의 Clickhouse Node 와 3 개의 Clickhouse Keeper 가 동작하게 됩니다.

ReplicatedMergeTree 생성하기.

아래의 SQL 쿼리를 통해서 ReplicatedMergeTree 테이블을 생성할 수 있습니다.

일반 MergeTree Table 과의 차이점은 ON CLUSTER 선언문을 사용한다는 점입니다.

반드시 어떤 클러스터에 테이블을 적용하는지 명시해주셔야 합니다.

create table default.user_actions ON CLUSTER cluster_1S_2R (
    user String,
    action String,
    acted_at DateTime
) engine=ReplicatedMergeTree
order by acted_at
partition by toYYYYMM(acted_at);

Query id: 06477320-4549-41e6-a9d8-717e30200789

┌─host──────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ clickhouse-01 │ 9000 │      0 │       │                   1 │                0 │
│ clickhouse-02 │ 9000 │      0 │       │                   0 │                0 │
└───────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

2 rows in set. Elapsed: 0.131 sec.

< 데이터 생성하기 >

insert into default.user_actions(user, action, acted_at)
values ('Andy', 'Visit', '2023-01-01 00:00:00'),
('Brian', 'Buy', '2023-01-01 00:01:00'),
('Dennis', 'View', '2023-01-01 00:02:00'),
('Chris', 'View', '2023-01-01 00:03:00'),
('Emma', 'Buy', '2023-01-01 00:04:00'),
('Fabian', 'Cart', '2023-01-01 00:05:00'),
('Gareth', 'RemoveCart', '2023-01-01 00:06:00'),
('Hayden', 'Cart', '2023-01-01 00:07:00'),
('Illa', 'View', '2023-01-01 00:08:00')

Query id: 24ec5491-75a8-4bab-adbb-5d27f0b8cdf5

Ok.

9 rows in set. Elapsed: 0.034 sec.

< Replication 확인하기 >

clickhouse-01, clickhouse-02 노드 모두에 데이터가 올바르게 생성되고 복제되었는지를 확인합니다.

select * from user_actions;

SELECT *
FROM user_actions

Query id: cc600211-8a38-47ea-a428-5fcd4699c030

┌─user───┬─action─────┬────────────acted_at─┐
│ Andy   │ Visit      │ 2023-01-01 00:00:00 │
│ Brian  │ Buy        │ 2023-01-01 00:01:00 │
│ Dennis │ View       │ 2023-01-01 00:02:00 │
│ Chris  │ View       │ 2023-01-01 00:03:00 │
│ Emma   │ Buy        │ 2023-01-01 00:04:00 │
│ Fabian │ Cart       │ 2023-01-01 00:05:00 │
│ Gareth │ RemoveCart │ 2023-01-01 00:06:00 │
│ Hayden │ Cart       │ 2023-01-01 00:07:00 │
│ Illa   │ View       │ 2023-01-01 00:08:00 │
└────────┴────────────┴─────────────────────┘

9 rows in set. Elapsed: 0.011 sec.

select * from remote('clickhouse-02:9000', 'default.user_actions');

SELECT *
FROM remote('clickhouse-02:9000', 'default.user_actions')

Query id: 70335f4f-6867-4164-ad6e-d15b4aec23f9

┌─user───┬─action─────┬────────────acted_at─┐
│ Andy   │ Visit      │ 2023-01-01 00:00:00 │
│ Brian  │ Buy        │ 2023-01-01 00:01:00 │
│ Dennis │ View       │ 2023-01-01 00:02:00 │
│ Chris  │ View       │ 2023-01-01 00:03:00 │
│ Emma   │ Buy        │ 2023-01-01 00:04:00 │
│ Fabian │ Cart       │ 2023-01-01 00:05:00 │
│ Gareth │ RemoveCart │ 2023-01-01 00:06:00 │
│ Hayden │ Cart       │ 2023-01-01 00:07:00 │
│ Illa   │ View       │ 2023-01-01 00:08:00 │
└────────┴────────────┴─────────────────────┘

9 rows in set. Elapsed: 0.016 sec.

'Database > Clickhouse' 카테고리의 다른 글

[Clickhouse] Docker 로 Clickhouse 구현하기 (2)	2024.02.17
[ClickHouse] Multi Sharding 구현하기 ( Docker, Shards ) (0)	2024.02.17
[ClickHouse] Compact Wide Parts 알아보기 ( part_type ) (0)	2024.01.16
[ClickHouse] Block 알아보기 (0)	2024.01.10
ClickHouse MergeTree 알아보기 (0)	2023.11.07

ABOUT ME

코딩수집 코딩수집

- 목차

들어가며.

1 Shard & 1 Clickhouse-Keeper.

ch-1S_1K Clickhouse Cluster 실행하기.

MergeTree Table 생성하기.

1 Shard & 2 Replicas 구성하기.

ReplicatedMergeTree 생성하기.

'Database > Clickhouse' 카테고리의 다른 글

티스토리툴바

ABOUT ME

- 목차

들어가며.

1 Shard & 1 Clickhouse-Keeper.

ch-1S_1K Clickhouse Cluster 실행하기.

MergeTree Table 생성하기.

1 Shard & 2 Replicas 구성하기.

ReplicatedMergeTree 생성하기.

'Database > Clickhouse' 카테고리의 다른 글

관련글 관련글 더보기

티스토리툴바