# clickhouse/clickhouse-go architecture

> Architecture diagram of the GitHub repository clickhouse/clickhouse-go, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/clickhouse/clickhouse-go
- Repository: https://github.com/clickhouse/clickhouse-go

## Overview

##### Project Type and Purpose
- This is an open-source Go library (“clickhouse-go”) offering two primary interfaces:
– Native TCP protocol client
– database/sql (“std”) client with optional HTTP transport
- It provides connection pooling, encoding/decoding via the ch-go engine, compression, settings, OpenTelemetry, async-insert, and clickhouse-specific features (client-info, external data, etc.).

##### File Structure
- Top-level API package:
– bind.go, conn_*.go, clickhouse.go, context.go, client_info.go
- Low-level modules under lib/ and lib/proto/:
– lib/binary, lib/column (code-generated column types), lib/proto for network wire formats, lib/timezone, lib/driver
- Example clients: examples/clickhouse_api and examples/std
- Benchmarks: benchmark/v1 & v2
- Tests: end-to-end in tests/, tests/std/, stress tests
- Internal tool: internal/cmd/release (CLI for packaging/release)
- CI/CD and deployment: .github/workflows, .github/cloud/service.tf, Docker compose

##### Extract Architecture Insights from README
- Two interface layers (Native & database/sql) share common options, settings, compression, and protocol modules.
- Uses ch-go for optimized columnar I/O.
- Supports TCP and optional HTTP transport (over native or HTTP protocols).
- Advanced query features via Context (settings, query parameters, telemetry).

##### Main Components and Their Relationships
A. API Layer
1. Native API (clickhouse.Open, Conn methods)
2. Std API (clickhouse.OpenDB, implements database/sql/driver)
B. Connection Manager
– Connection pool, open strategy, max idle/open, lifetime
– Context watchdog, ping, profiling
C. Protocol Layer
– TCP handlers: conn_handshake.go, conn_query.go, conn_send_query.go, conn_profile_events.go
– HTTP handlers: conn_http_*.go, HTTP proxy support
D. Encoding/Decoding & Compression
– ch-go subclient (lib/proto, lib/binary) for columnar block format
– Compression modules (LZ4, ZSTD) integrated at block level
– Code-generated column types in lib/column
E. Settings & Client Info
– Settings struct, Query parameters, ClientInfo metadata
F. AsyncInsert & Batch API
– PrepareBatch, AsyncInsert methods, batch.go, conn_batch.go
G. External Dependencies
– ClickHouse server (TCP port 9000, HTTP port 8123/8443)
– TLS stack (crypto/tls), optional HTTP proxy, environment variables
– database/sql package, net.Dialer

##### Architectural Patterns and Principles
- Layered Abstraction:
– Top: user-friendly API (native vs std)
– Middle: connection and protocol management
– Bottom: binary/columnar encoding + network I/O
- Pluggable Transports: TCP vs HTTP via Protocol interface
- Code-generation for performance (column types)
- Context-driven query controls (telemetry, settings)
- Separation of concerns: examples/ & benchmark/ folders isolated from core library

##### Detailed Capturing of Essential Elements
- Dual-interface branching (native vs std) early in flow.
- Optional paths (HTTP transport, AsyncInsert path).
- Batch vs immediate query execution.
- External utilities:
– examples/ (usage), tests/ (validation), CI/CD pipelines, Docker compose

## Diagram (Mermaid)

```mermaid
flowchart TD
    UA["User Application"]:::external

    subgraph "API Layer"
        NAPI["Native API"]:::api
        STDAPI["Std API (database/sql)"]:::api
    end

    SETTINGS["Settings & ClientInfo"]:::config

    UA --> NAPI
    UA --> STDAPI
    SETTINGS --> NAPI
    SETTINGS --> STDAPI

    subgraph "AsyncInsert & Batch API"
        AI["AsyncInsert"]:::async
        BATCH["Batch"]:::async
    end

    NAPI --> AI
    AI --> BATCH
    BATCH --> CM

    subgraph "Connection Manager"
        CM["Connection Manager"]:::manager
    end

    NAPI --> CM
    STDAPI --> CM

    CM --> PROTO_DECIDE{"Transport?"}:::decision

    PROTO_DECIDE -->|TCP| T1
    PROTO_DECIDE -->|HTTP| H1

    subgraph "TCP Protocol Handlers"
        T1["Handshake"]:::protocol
        T2["Query"]:::protocol
        T3["SendQuery"]:::protocol
        T4["ProfileEvents"]:::protocol
    end

    T1 --> T2 --> T3 --> T4

    subgraph "HTTP Protocol Handlers"
        H1["HTTP Conn"]:::protocol
        H2["HTTP Query"]:::protocol
        H3["HTTP Exec"]:::protocol
        H4["HTTP Batch"]:::protocol
        H5["HTTP AsyncInsert"]:::protocol
    end

    H1 --> H2 --> H3 --> H4 --> H5

    T4 --> EE1
    H5 --> EE1

    subgraph "Encoding/Decoding Engine"
        EE1["Block"]:::engine
        EE2["Proto Handshake"]:::engine
        EE3["Proto Query"]:::engine
        EB1["Binary String"]:::engine
        EB2["String Safe"]:::engine
        EB3["String Unsafe"]:::engine
        subgraph "Column Codegen Types"
            CCG["lib/column"]:::engine
        end
        Compression["Compression Modules"]:::engine
    end

    EE1 --> EE2
    EE2 --> EE3
    EE3 --> EB1
    EB1 --> EB2 --> EB3
    EB3 --> CCG
    CCG --> Compression
    Compression --> NetworkIO["Network I/O"]:::external
    NetworkIO --> CH["ClickHouse Server"]:::external

    CH --> EE1
    EE1 --> CM
    CM --> NAPI
    CM --> STDAPI
    NAPI --> UA
    STDAPI --> UA

    subgraph "Examples & Benchmarks"
        EX1["examples/clickhouse_api"]:::external
        EX2["examples/std"]:::external
        BM1["benchmark/v1"]:::external
        BM2["benchmark/v2"]:::external
    end

    subgraph "Tests & CI"
        TST1["tests/"]:::external
        TST2["tests/std/"]:::external
        TST3["tests/stress/"]:::external
        CI1[".github/workflows/"]:::external
        CI2[".github/cloud/service.tf"]:::external
        DC["docker-compose.yml"]:::external
        REL["Internal Release Tool"]:::external
    end

    click NAPI "https://github.com/clickhouse/clickhouse-go/blob/main/clickhouse.go"
    click STDAPI "https://github.com/clickhouse/clickhouse-go/blob/main/clickhouse_std.go"
    click SETTINGS "https://github.com/clickhouse/clickhouse-go/blob/main/clickhouse_options.go"
    click CM "https://github.com/clickhouse/clickhouse-go/blob/main/conn.go"
    click T1 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_handshake.go"
    click T2 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_query.go"
    click T3 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_send_query.go"
    click T4 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_profile_events.go"
    click H1 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_http.go"
    click H2 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_http_query.go"
    click H3 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_http_exec.go"
    click H4 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_http_batch.go"
    click H5 "https://github.com/clickhouse/clickhouse-go/blob/main/conn_http_async_insert.go"
    click EE1 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/proto/block.go"
    click EE2 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/proto/handshake.go"
    click EE3 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/proto/query.go"
    click EB1 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/binary/string.go"
    click EB2 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/binary/string_safe.go"
    click EB3 "https://github.com/clickhouse/clickhouse-go/blob/main/lib/binary/string_unsafe.go"
    click CCG "https://github.com/clickhouse/clickhouse-go/tree/main/lib/column"
    click Compression "https://github.com/clickhouse/clickhouse-go/blob/main/ext/ext.go"
    click AI "https://github.com/clickhouse/clickhouse-go/blob/main/conn_async_insert.go"
    click BATCH "https://github.com/clickhouse/clickhouse-go/blob/main/conn_batch.go"
    click EX1 "https://github.com/clickhouse/clickhouse-go/tree/main/examples/clickhouse_api/"
    click EX2 "https://github.com/clickhouse/clickhouse-go/tree/main/examples/std/"
    click BM1 "https://github.com/clickhouse/clickhouse-go/tree/main/benchmark/v1/"
    click BM2 "https://github.com/clickhouse/clickhouse-go/tree/main/benchmark/v2/"
    click TST1 "https://github.com/clickhouse/clickhouse-go/tree/main/tests/"
    click TST2 "https://github.com/clickhouse/clickhouse-go/tree/main/tests/std/"
    click TST3 "https://github.com/clickhouse/clickhouse-go/tree/main/tests/stress/"
    click CI1 "https://github.com/clickhouse/clickhouse-go/tree/main/.github/workflows/"
    click CI2 "https://github.com/clickhouse/clickhouse-go/blob/main/.github/cloud/service.tf"
    click DC "https://github.com/clickhouse/clickhouse-go/blob/main/docker-compose.yml"
    click REL "https://github.com/clickhouse/clickhouse-go/blob/main/internal/cmd/release/main.go"

    classDef api fill:#f9f,stroke:#333,stroke-width:1px
    classDef manager fill:#9f9,stroke:#333,stroke-width:1px
    classDef protocol fill:#ff9,stroke:#333,stroke-width:1px
    classDef engine fill:#9ff,stroke:#333,stroke-width:1px
    classDef external fill:#ccc,stroke:#333,stroke-width:1px
    classDef config fill:#fc9,stroke:#333,stroke-width:1px
    classDef async fill:#c9f,stroke:#333,stroke-width:1px
    classDef decision fill:#f96,stroke:#333,stroke-width:1px,shape:diamond
```
