# vkcom/statshouse architecture

> Architecture diagram of the GitHub repository vkcom/statshouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vkcom/statshouse
- Repository: https://github.com/vkcom/statshouse

## Overview

##### Main Components Overview

a) Data Collection Layer:
- Agent components (receiving metrics from clients)
- Aggregator components (processing and forwarding metrics)
- Multiple client libraries (Go, PHP, C++, Java, Python, nginx)

b) Storage Layer:
- ClickHouse cluster as the primary data store
- Multiple resolution storage (1s, 1m, 1h)
- Data sampling and budgeting system

c) Query and Visualization Layer:
- StatsHouse API
- Built-in UI
- Grafana plugin interface
- Metadata service

##### Component Relationships

**Clients -> Agents -> Aggregators -> ClickHouse Cluster**
- ↑
- API --------------┘
- ↑
- UI/Grafana

##### Specific Implementation Details

Technical elements:

a) Data Flow:
- UDP metric ingestion path
- Sampling and budgeting checkpoints
- Automatic downsampling flows
- Metadata synchronization paths

b) High Availability Features:
- Multiple agent instances
- ClickHouse cluster replication
- Load balancing components

c) Scaling Components:
- Sharding boundaries
- Multi-tenant isolation
- Budget enforcement points

##### Additional Elements

Supporting details:
- Metric resolution tiers (1s, 1m, 1h)
- Data retention policies
- Network protocols (UDP, TCP)
- API endpoints
- Monitoring paths (self-monitoring)

##### Operational Aspects

Operational features:
- Automatic failover paths
- Sampling mechanisms
- Budget enforcement points
- Metadata synchronization
- Configuration distribution

##### Key Metrics

Measurement points for:
- 1.2B metrics per second ingestion
- 22,000 server support
- 5-year data retention
- 5-second latency target

- High availability
- Scalability
- Multi-tenancy
- Automatic downsampling
- Budget enforcement
- Failure resilience

This design should reflect StatsHouse's production-proven architecture while highlighting its unique features like automatic sampling, budget enforcement, and multi-resolution storage.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Collection Layer"
        CL["Client Libraries"]:::client
        AG["Agent"]:::agent
        AGG["Aggregator"]:::aggregator
    end

    subgraph "Storage Layer"
        CH[("ClickHouse Cluster")]:::storage
        subgraph "Resolution Tiers"
            R1["1s Resolution"]:::storage
            R2["1m Resolution"]:::storage
            R3["1h Resolution"]:::storage
        end
    end

    subgraph "Query & Visualization Layer"
        API["StatsHouse API"]:::api
        UI["Built-in UI"]:::ui
        GF["Grafana Plugin"]:::ui
        MD["Metadata Service"]:::service
    end

    subgraph "Data Processing"
        DM["Data Model"]:::process
        DS["Data Sampling"]:::process
        MM["Metric Mapping"]:::process
        PR["Protocol Handler"]:::process
        CF["Config Manager"]:::process
    end

    CL --> |"UDP Metrics"|AG
    AG --> |"Forward"|AGG
    AGG --> |"Process & Store"|CH
    CH --> |"Query Data"|API
    API --> |"Serve Data"|UI
    API --> |"Serve Data"|GF
    MD --> |"Metadata Sync"|API
    MD --> |"Metadata Sync"|AGG
    DM --> |"Structure"|AGG
    DS --> |"Sample"|AGG
    MM --> |"Map"|AGG
    PR --> |"Protocol"|AG
    CF --> |"Configure"|AG
    CF --> |"Configure"|AGG

    %% Component Mappings
    click CL "https://github.com/VKCOM/statshouse/tree/master/cmd/statshouse-client-test/"
    click AG "https://github.com/VKCOM/statshouse/tree/master/internal/agent/"
    click AGG "https://github.com/VKCOM/statshouse/tree/master/internal/aggregator/"
    click CH "https://github.com/VKCOM/statshouse/blob/master/build/clickhouse-cluster.sql"
    click DM "https://github.com/VKCOM/statshouse/tree/master/internal/data_model/"
    click API "https://github.com/VKCOM/statshouse/tree/master/cmd/statshouse-api/"
    click UI "https://github.com/VKCOM/statshouse/tree/master/statshouse-ui/"
    click GF "https://github.com/VKCOM/statshouse/tree/master/grafana-plugin-ui/"
    click MD "https://github.com/VKCOM/statshouse/tree/master/cmd/statshouse-metadata/"
    click DS "https://github.com/VKCOM/statshouse/blob/master/internal/data_model/sampling.go"
    click MM "https://github.com/VKCOM/statshouse/tree/master/internal/mapping/"
    click PR "https://github.com/VKCOM/statshouse/blob/master/internal/data_model/proto.go"
    click CF "https://github.com/VKCOM/statshouse/tree/master/internal/config/"

    classDef client fill:#B2D8B2
    classDef agent fill:#9CC1E5
    classDef aggregator fill:#9CC1E5
    classDef storage fill:#85C1E9
    classDef api fill:#F9E79F
    classDef ui fill:#F5CBA7
    classDef service fill:#D2B4DE
    classDef process fill:#AEB6BF

    %% Legend
    subgraph Legend
        L1["Client Components"]:::client
        L2["Ingestion Components"]:::agent
        L3["Storage Components"]:::storage
        L4["API Components"]:::api
        L5["UI Components"]:::ui
        L6["Services"]:::service
        L7["Processing"]:::process
    end
```
