# hoptical/grafana-kafka-datasource architecture

> Architecture diagram of the GitHub repository hoptical/grafana-kafka-datasource, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hoptical/grafana-kafka-datasource
- Repository: https://github.com/hoptical/grafana-kafka-datasource

## Overview

##### Main Components
- Frontend Plugin Component:
- Located primarily in the src directory (e.g., ConfigEditor.tsx, QueryEditor.tsx, module.ts).
- Contains assets like images and configuration files (plugin.json) needed for the Grafana UI.
- Backend Plugin Component:
- Written in Go and found mostly under the pkg directory (e.g., pkg/main.go, pkg/kafka_client/client.go, pkg/plugin).
- Uses Magefile.go for build automation and configuration management.
- Configuration, Build, and Testing Infrastructure:
- .config directory holds settings for ESLint, Prettier, Jest, Docker, Webpack, and other build or test configurations.
- Additional project-level files: docker-compose.yaml (suggesting containerized environments for running Grafana/Kafka), go.mod, and package.json run the dependency management.
- External Dependencies / Environment:
- Grafana Server: The plugin is installed via grafana-cli or manual installation into the Grafana plugins directory.
- Kafka Broker: The datasource accesses the Kafka broker to query topics and stream data.
- Example Producer (provided under example/python) shows a sample integration which suggests how data enters the Kafka topics.

##### Relationships and Data Flow
- The Frontend (React/TypeScript components) provide a configuration and query editor interface which is part of Grafana’s dashboard, allowing users to set things like Kafka server URLs and query parameters.
- The Backend (written in Go using the Grafana Plugin SDK) interacts with Kafka:
- Receives configuration details from the frontend.
- Connects to the Kafka broker using the confluent-kafka-go library.
- Consumes messages from Kafka topics and sends data back to Grafana for visualization.
- The external data flow:
- The Grafana Server loads the plugin.
- Plugins communicate with the Kafka broker.
- Example scripts (in the example folder) can act as producers sending data to Kafka topics.
- Build and Deployment:
- Tools like Docker (Dockerfile in .config and docker-compose.yaml) indicate containerized deployment or testing environments.
- Configuration tools like Webpack and Jest help build and test the frontend, while Mage is used for building the Go backend.

##### Important Architectural Patterns and Technologies
- Separation of Concerns: Frontend (UI/UX for data source settings and query parameters) is cleanly separated from the backend (data processing and Kafka interaction).
- Plugin Architecture: Designed as a Grafana plugin, it adheres to the Grafana plugin SDK’s guidelines and structure.
- Build & Test Frameworks: Use of Jest for testing, Webpack for bundling, and Mage for backend builds indicates a modern, automated build pipeline.
- Containerization: Docker-related configurations point towards deployment and testing in containerized environments.
- Language/Frameworks:
- Go for backend with the Grafana Plugin SDK.
- TypeScript/React for the frontend.
- Configuration and testing infrastructure via ESLint, Prettier, and Jest.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        GS["Grafana Server"]:::external
        KB["Kafka Broker"]:::external
    end

    %% Grafana Plugin Frontend (React/TypeScript)
    subgraph "Grafana Plugin Frontend (React/TypeScript)"
        FE1["ConfigEditor.tsx"]:::frontend
        FE2["QueryEditor.tsx"]:::frontend
        FE3["module.ts"]:::frontend
        FE4["plugin.json"]:::frontend
    end

    %% Grafana Plugin Backend (Go)
    subgraph "Grafana Plugin Backend (Go)"
        BE1["main.go"]:::backend
        BE2["Kafka Client"]:::backend
        BE3["Plugin Logic"]:::backend
        BE4["Plugin Test"]:::backend
    end

    %% Build and Automation Infrastructure
    subgraph "Build and Automation Infrastructure"
        INF1["Magefile.go"]:::infra
        INF2["docker-compose.yaml"]:::infra
        INF3[".config/Dockerfile"]:::infra
    end

    %% Configuration, Linting, Testing & Bundling
    subgraph "Configuration, Linting, Testing & Bundling"
        CONF1[".config"]:::infra
        CONF2[".config/jest.config.js"]:::infra
        CONF3[".config/jest-setup.js"]:::infra
        CONF4["Webpack Config"]:::infra
    end

    %% Example / Data Producer
    subgraph "Example/Data Producer for Kafka"
        EX1["sample_producer.py"]:::infra
        EX2["Example README"]:::infra
    end

    %% Data Flow & Interactions
    GS -->|"load_plugin"| FE4
    FE4 -->|"config_update"| BE3
    FE1 -->|"user_config"| FE4
    FE2 -->|"build_query"| FE4

    BE1 -->|"initiate_plugin"| BE3
    BE3 -->|"use_client"| BE2
    BE2 -->|"kafka_connect"| KB

    EX1 -->|"send_data"| KB

    INF1 ---|"build_backend"| BE1
    INF1 ---|"build_backend"| BE3
    INF2 ---|"deploy_env"| GS
    INF2 ---|"deploy_env"| KB
    INF3 ---|"docker_build"| INF2

    CONF1 ---|"lint/test/build"| FE1
    CONF1 ---|"lint/test/build"| BE1
    CONF2 ---|"jest_test"| FE2
    CONF3 ---|"jest_setup"| FE2
    CONF4 ---|"bundle_assets"| FE3

    %% Click Events for Frontend Components
    click FE1 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/src/ConfigEditor.tsx"
    click FE2 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/src/QueryEditor.tsx"
    click FE3 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/src/module.ts"
    click FE4 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/src/plugin.json"

    %% Click Events for Backend Components
    click BE1 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/pkg/main.go"
    click BE2 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/pkg/kafka_client/client.go"
    click BE3 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/pkg/plugin/plugin.go"
    click BE4 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/pkg/plugin/plugin_test.go"

    %% Click Events for Build and Automation Infrastructure
    click INF1 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/Magefile.go"
    click INF2 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/docker-compose.yaml"
    click INF3 "https://github.com/hoptical/grafana-kafka-datasource/tree/main/.config/Dockerfile"

    %% Click Events for Configuration, Linting, Testing & Bundling
    click CONF1 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/.config"
    click CONF2 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/.config/jest.config.js"
    click CONF3 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/.config/jest-setup.js"
    click CONF4 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/.config/webpack/webpack.config.ts"

    %% Click Events for Example/Data Producer
    click EX1 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/example/python/sample_producer.py"
    click EX2 "https://github.com/hoptical/grafana-kafka-datasource/blob/main/example/python/README.md"

    %% Styles
    classDef frontend fill:#f9d5e5,stroke:#333,stroke-width:1px;
    classDef backend fill:#d5e8d4,stroke:#333,stroke-width:1px;
    classDef external fill:#d0e0e3,stroke:#333,stroke-width:1px;
    classDef infra fill:#f9e79f,stroke:#333,stroke-width:1px;
```
