# tencentblueking/bkmonitor-datalink architecture

> Architecture diagram of the GitHub repository tencentblueking/bkmonitor-datalink, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencentblueking/bkmonitor-datalink
- Repository: https://github.com/tencentblueking/bkmonitor-datalink

## Overview

##### Project type and purpose
- This is not a monolithic full-stack app but a suite of Go-based microservices and SDKs that together form a “data link” pipeline for the BlueKing observability platform. It provides:
- Data collection (metrics & events)
- Third-party alert ingestion
- Data transformation and routing
- High-availability proxy to InfluxDB
- A unified PromQL-style query service
- Task scheduling and control via a Kubernetes operator

##### File structure (pkg/ directory)
- Top-level service modules:
- bkmonitorbeat – host-side collector agent (Beats-style)
- ingester – HTTP/Kafka pull or push of alert events
- transfer – core ETL: convert data formats and publish to storage back ends
- influxdb-proxy – HA proxy & load-balancer for InfluxDB clusters
- unify-query – HTTP/gRPC query service providing PromQL/STQL over InfluxDB
- bk-monitor-worker – job scheduler & worker pool for periodic or event tasks
- bkmonitor-operator – Kubernetes Operator (CRDs, controllers, Helm)
- libgse & kits – shared SDKs and helper libraries
- Each module is a standalone Go microservice (cmd/ or main.go), uses dependencies (Redis, Kafka, Consul, HTTP, gRPC), and ships its own Dockerfile/Helm chart.

##### README for architecture notes
- Design goals: plug-and-play modules, decoupled via config, MQ, standard protocols

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Collection Layer
    subgraph "Collection Layer"
        direction TB
        DataCollector["bkmonitorbeat"]:::data-plane
        Ingester["ingester"]:::data-plane
    end

    %% Transformation Layer
    subgraph "Transformation Layer"
        direction TB
        Transfer["transfer (ETL)"]:::data-plane
        PluginSlots["Plugins: logs, metrics, traces"]:::data-plane
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        Kafka["Kafka"]:::external
        Redis["Redis"]:::external
        InfluxProxy["influxdb-proxy"]:::data-plane
        InfluxDB["InfluxDB"]:::external
    end

    %% Query Layer
    subgraph "Query Layer"
        direction TB
        UnifyQuery["unify-query (PromQL/STQL API)"]:::data-plane
    end

    %% Control Plane
    subgraph "Control Plane"
        direction TB
        Operator["bkmonitor-operator"]:::control-plane
        Worker["bk-monitor-worker"]:::control-plane
        KubernetesAPI["Kubernetes API"]:::external
    end

    %% SDK Layer
    subgraph "SDK Layer"
        direction TB
        LibGSE["libgse"]:::sdk
    end

    %% Data Flows
    DataCollector -->|"HTTP/Kafka"| Transfer
    Ingester -->|"HTTP/Kafka"| Transfer
    Transfer --> Kafka
    Transfer --> Redis
    Kafka --> InfluxProxy
    Redis --> InfluxProxy
    InfluxProxy -->|"HTTP/gRPC"| InfluxDB
    Transfer -->|"metadata/events"| Worker
    UnifyQuery -->|"HTTP/gRPC"| InfluxProxy

    %% Control Flows
    Operator -->|"watches CRDs"| KubernetesAPI
    Operator -- "creates tasks" --> Worker
    Worker -- "executes jobs" --> Transfer
    Worker -- "triggers ingestion" --> Ingester
    Worker -- "controls collectors" --> DataCollector

    %% SDK usage
    LibGSE --- Transfer
    LibGSE --- DataCollector
    LibGSE --- Ingester
    LibGSE --- UnifyQuery
    LibGSE --- Operator
    LibGSE --- Worker

    %% Click Events
    click DataCollector "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/bkmonitorbeat"
    click Ingester "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/ingester"
    click Transfer "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/transfer"
    click InfluxProxy "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/influxdb-proxy"
    click UnifyQuery "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/unify-query"
    click Worker "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/bk-monitor-worker"
    click Operator "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/operator"
    click LibGSE "https://github.com/tencentblueking/bkmonitor-datalink/tree/master/pkg/libgse"

    %% Styles
    classDef data-plane fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef control-plane fill:#90EE90,stroke:#000,stroke-width:1px
    classDef sdk fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
```
