How greptimeteam/greptimedb works
Project Type and Purpose
• GreptimeDB is a cloud‑native, distributed observability database (Metrics, Logs, Traces) written in Rust.
• It implements a separated compute–storage architecture, offers SQL & PromQL, stream processing (Flow), and integrates with object stores (S3, Azure Blob, etc.) and external protocols (MySQL, PostgreSQL, HTTP/gRPC, OpenTelemetry).
High‑Level Architecture Components
a. Client Layer (Ingress)
• CLI (src/cli, src/cmd)
• SDKs & Ingesters (Go, Java, Rust, JS)
• Protocol endpoints in “servers” (src/servers):
– HTTP/REST Dashboard & APIs
– gRPC (Flight, Prometheus Gateway, OTLP)
– MySQL & PostgreSQL wire protocols
– OpenMetrics/Prometheus remote‑read/write
– Ingestion pipeline (Pipeline crate) for log/metric/event transforms
b. Frontend Layer (src/frontend)
• Parses SQL/PromQL, handles query planning & coordination
• Provides rate limiting, authentication (src/auth), and quota enforcement (src/limiter)
• Interacts with Meta Service for routing and with Compute nodes for distributed execution
c. Meta Service (src/meta-srv, src/meta-client)
• Leader election, cluster metadata, region/table assignment
• State persisted in etcd or pluggable KV backends (common/meta/src/kv_backend)
• Exposes RPC API for heartbeats, routing, DDL, procedures
d. Compute Engines
• Stream/Flow Engine (src/flow) – stream processing tasks with backpressure & windowing
• PromQL Engine (src/promql) – time‑series range & instant queries
e. Storage Nodes
• Datanode (src/datanode) – region servers that store data in local file‑engine or object store
• Metric‑Engine (src/metric-engine) – time‑series ingestion, compaction, TTL
• File‑Engine (src/file-engine) – columnar file format, SST files, manifest management
• Log‑Store (src/log-store) – Kafka‑backed log ingestion and index sink
• Index module (src/index) – inverted, full‑text, bloom filters
f. Underlying Libraries
• Common crates under src/common/* (config, error, runtime, telemetry, wal, datasource, catalog, etc.)
• Object Store abstraction (src/object-store) – S3, Azure Blob, GCS, FS
• WAL & manifest layers (common/wal, puffin)
g. External Services
• Object Storage (AWS S3, Azure Blob, MinIO)
• etcd (cluster metadata)
• Kafka (optional log stream)
• SQL clients & Grafana for visualization
Capturing Architectural Patterns
• Control plane / data plane separation: meta‑srv vs. datanode
• Pluggable backends (KV, object store, protocols) – highlight as interfaces
• Distributed query planning and execution – show coordinator (frontend) + workers