# greptimeteam/greptimedb architecture

> Architecture diagram of the GitHub repository greptimeteam/greptimedb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/greptimeteam/greptimedb
- Repository: https://github.com/greptimeteam/greptimedb

## Overview

##### Project Type and Purpose
- GreptimeDB is a cloud‑native, distributed observability database (Metrics, Logs, Traces) written in Rust.
- It implements a separated compute–storage architecture, offers SQL & PromQL, stream processing (Flow), and integrates with object stores (S3, Azure Blob, etc.) and external protocols (MySQL, PostgreSQL, HTTP/gRPC, OpenTelemetry).

##### High‑Level Architecture Components
a. Client Layer (Ingress)
- CLI (`src/cli`, `src/cmd`)
- SDKs & Ingesters (Go, Java, Rust, JS)
- Protocol endpoints in “servers” (`src/servers`):
– HTTP/REST Dashboard & APIs
– gRPC (Flight, Prometheus Gateway, OTLP)
– MySQL & PostgreSQL wire protocols
– OpenMetrics/Prometheus remote‑read/write
– Ingestion pipeline (Pipeline crate) for log/metric/event transforms

b. Frontend Layer (`src/frontend`)
- Parses SQL/PromQL, handles query planning & coordination
- Provides rate limiting, authentication (`src/auth`), and quota enforcement (`src/limiter`)
- Interacts with Meta Service for routing and with Compute nodes for distributed execution

c. Meta Service (`src/meta-srv`, `src/meta-client`)
- Leader election, cluster metadata, region/table assignment
- State persisted in etcd or pluggable KV backends (`common/meta/src/kv_backend`)
- Exposes RPC API for heartbeats, routing, DDL, procedures

d. Compute Engines
- Stream/Flow Engine (`src/flow`) – stream processing tasks with backpressure & windowing
- PromQL Engine (`src/promql`) – time‑series range & instant queries

e. Storage Nodes
- Datanode (`src/datanode`) – region servers that store data in local file‑engine or object store
- Metric‑Engine (`src/metric-engine`) – time‑series ingestion, compaction, TTL
- File‑Engine (`src/file-engine`) – columnar file format, SST files, manifest management
- Log‑Store (`src/log-store`) – Kafka‑backed log ingestion and index sink
- Index module (`src/index`) – inverted, full‑text, bloom filters

f. Underlying Libraries
- Common crates under `src/common/*` (config, error, runtime, telemetry, wal, datasource, catalog, etc.)
- Object Store abstraction (`src/object-store`) – S3, Azure Blob, GCS, FS
- WAL & manifest layers (`common/wal`, `puffin`)

g. External Services
- Object Storage (AWS S3, Azure Blob, MinIO)
- etcd (cluster metadata)
- Kafka (optional log stream)
- SQL clients & Grafana for visualization

##### Capturing Architectural Patterns
- Control plane / data plane separation: meta‑srv vs. datanode
- Pluggable backends (KV, object store, protocols) – highlight as interfaces
- Distributed query planning and execution – show coordinator (frontend) + workers

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer" 
        direction TB
        CLI["CLI"]:::client
        SDK["SDKs & Ingesters"]:::client
        Protocol["Protocol Endpoints"]:::client
    end

    %% Frontend Layer
    subgraph "Frontend Layer" 
        direction TB
        Parser["SQL/PromQL Parser & Coordinator"]:::frontend
        Auth["Authentication"]:::frontend
        Quota["Rate Limiting/Quota"]:::frontend
    end

    %% Meta Service
    subgraph "Meta Service (Control Plane)" 
        direction TB
        MetaSrv["Meta Service Server"]:::control
        MetaClient["Meta Service Client"]:::control
        KVBackend["Pluggable KV Backends"]:::control
    end

    %% Compute Plane
    subgraph "Compute Plane" 
        direction TB
        QueryEng["Query Engine"]:::compute
        FlowEng["Stream/Flow Engine"]:::compute
        PromQLEng["PromQL Engine"]:::compute
    end

    %% Storage Nodes (Data Plane)
    subgraph "Storage Nodes" 
        direction TB
        Datanode["Datanode"]:::storage
        MetricEng["Metric-Engine"]:::storage
        FileEng["File-Engine"]:::storage
        LogStore["Log-Store"]:::storage
        IndexMod["Index Module"]:::storage
    end

    %% External Services
    subgraph "External Services" 
        direction TB
        Etcd["etcd"]:::external
        ObjectStore["Object Store (S3/Azure/GCS)"]:::external
        Kafka["Kafka"]:::external
        Grafana["Grafana & SQL Clients"]:::external
    end

    %% Connections
    CLI -->|"ingest/query"| Parser
    SDK -->|"ingest/query"| Parser
    Protocol -->|"HTTP/gRPC/MySQL"| Parser

    Parser -->|"route lookup"| MetaSrv
    Auth --> Parser
    Quota --> Parser

    Parser -->|"distributed query"| QueryEng
    Parser -->|"stream tasks"| FlowEng
    Parser -->|"time-series query"| PromQLEng

    QueryEng -->|"region read/write"| Datanode
    FlowEng -->|"read/write"| MetricEng
    PromQLEng -->|"read SST"| FileEng

    Datanode -->|"heartbeat"| MetaSrv
    Datanode -->|"persist SST/WAL"| ObjectStore
    MetricEng -->|"persist data"| ObjectStore
    FileEng -->|"store files"| ObjectStore
    LogStore -->|"ingest"| Kafka
    LogStore -->|"index sink"| IndexMod

    MetaSrv -->|"store metadata"| Etcd
    MetaClient --> MetaSrv

    Grafana --> Parser

    %% Click Events
    click CLI "https://github.com/greptimeteam/greptimedb/tree/main/src/cli/"
    click SDK "https://github.com/greptimeteam/greptimedb/tree/main/src/client/"
    click Protocol "https://github.com/greptimeteam/greptimedb/tree/main/src/servers/"
    click Parser "https://github.com/greptimeteam/greptimedb/tree/main/src/frontend/"
    click Auth "https://github.com/greptimeteam/greptimedb/tree/main/src/auth/"
    click Quota "https://github.com/greptimeteam/greptimedb/tree/main/src/limiter/"
    click MetaSrv "https://github.com/greptimeteam/greptimedb/tree/main/src/meta-srv/"
    click MetaClient "https://github.com/greptimeteam/greptimedb/tree/main/src/meta-client/"
    click KVBackend "https://github.com/greptimeteam/greptimedb/tree/main/src/common/meta/src/kv_backend/"
    click QueryEng "https://github.com/greptimeteam/greptimedb/tree/main/src/query/"
    click FlowEng "https://github.com/greptimeteam/greptimedb/tree/main/src/flow/"
    click PromQLEng "https://github.com/greptimeteam/greptimedb/tree/main/src/promql/"
    click Datanode "https://github.com/greptimeteam/greptimedb/tree/main/src/datanode/"
    click MetricEng "https://github.com/greptimeteam/greptimedb/tree/main/src/metric-engine/"
    click FileEng "https://github.com/greptimeteam/greptimedb/tree/main/src/file-engine/"
    click LogStore "https://github.com/greptimeteam/greptimedb/tree/main/src/log-store/"
    click IndexMod "https://github.com/greptimeteam/greptimedb/tree/main/src/index/"
    click CommonLib "https://github.com/greptimeteam/greptimedb/tree/main/src/common/"
    click ObjStore "https://github.com/greptimeteam/greptimedb/tree/main/src/object-store/"
    click WAL "https://github.com/greptimeteam/greptimedb/tree/main/src/common/wal/"
    click Puffin "https://github.com/greptimeteam/greptimedb/tree/main/src/puffin/"

    %% Styles
    classDef client fill:#D0E8FF,stroke:#0366d6,color:#0366d6,stroke-width:1px;
    classDef frontend fill:#E6F4EA,stroke:#28a745,color:#28a745,stroke-width:1px;
    classDef control fill:#E8F0FE,stroke:#1967D2,color:#1967D2,stroke-width:1px;
    classDef compute fill:#FFE5D9,stroke:#D9480F,color:#D9480F,stroke-width:1px;
    classDef storage fill:#FFF4D6,stroke:#F59E0B,color:#B45309,stroke-width:1px;
    classDef external fill:#E2E8F0,stroke:#64748B,color:#334155,stroke-width:1px;
```
