How vkcom/statshouse works
Main Components Overview
a) Data Collection Layer:
• Agent components (receiving metrics from clients)
• Aggregator components (processing and forwarding metrics)
• Multiple client libraries (Go, PHP, C++, Java, Python, nginx)
b) Storage Layer:
• ClickHouse cluster as the primary data store
• Multiple resolution storage (1s, 1m, 1h)
• Data sampling and budgeting system
c) Query and Visualization Layer:
• StatsHouse API
• Built-in UI
• Grafana plugin interface
• Metadata service
Component Relationships
Clients -> Agents -> Aggregators -> ClickHouse Cluster
• ↑
• API --------------┘
• ↑
• UI/Grafana
Specific Implementation Details
Technical elements:
a) Data Flow:
• UDP metric ingestion path
• Sampling and budgeting checkpoints
• Automatic downsampling flows
• Metadata synchronization paths
b) High Availability Features:
• Multiple agent instances
• ClickHouse cluster replication
• Load balancing components
c) Scaling Components:
• Sharding boundaries
• Multi-tenant isolation
• Budget enforcement points
Additional Elements
Supporting details:
• Metric resolution tiers (1s, 1m, 1h)
• Data retention policies
• Network protocols (UDP, TCP)
• API endpoints
• Monitoring paths (self-monitoring)
Operational Aspects
Operational features:
• Automatic failover paths
• Sampling mechanisms
• Budget enforcement points
• Metadata synchronization
• Configuration distribution
Key Metrics
Measurement points for:
• 1.2B metrics per second ingestion
• 22,000 server support
• 5-year data retention
• 5-second latency target
• High availability
• Scalability
• Multi-tenancy
• Automatic downsampling
• Budget enforcement
• Failure resilience
This design should reflect StatsHouse's production-proven architecture while highlighting its unique features like automatic sampling, budget enforcement, and multi-resolution storage.