How deckhouse/prompp works
The Deckhouse Prom++ project is a high-performance fork of Prometheus that retains full compatibility with its configuration, API endpoints, storage formats, and overall functionality. However, it introduces major optimizations by rewriting the most resource-intensive components (namely the in-memory block construction and WAL (Write-Ahead Log) management) in C++. This has the effect of drastically reducing memory consumption (up to 10x reduction) while still operating as a drop‐in replacement for Prometheus.
Main Components
• Ingestion & Scraping Module: Represented by directories such as “scrape”, “discovery”, and “notifier”. These modules handle receiving metric data from various targets, applying service discoveries (integrations with Kubernetes, AWS, etc.), and then forwarding or notifying as needed.
• WAL Management & TSDB Storage: A key part of Prom++ is its optimized time-series database (TSDB) and WAL subsystem. Look at directories such as “tsdb”, “storage”, and “pp” (which contains the C++ optimized components including WAL handling and block construction). This is the engine that writes, reads, compacts, and serves stored metrics.
• Query Engine & PromQL Parser: The “promql” directory (with its engine, parser, and testing files), together with parts of “model”, encapsulate the logic for parsing and executing PromQL queries.
• API, Web UI & HTTP Server: The “web” directory, along with parts of “cmd/prometheus”, support API endpoints, the HTTP UI, and federation features. This component serves queries and configuration endpoints.
• Migration & Conversion Tools: The “cmd/prompptool” and “cmd/promtool” directories contain command-line utilities that automate WAL conversion, enabling seamless migration between Prometheus’s original and Prom++ formats.
• Configuration and Rules Engine: Directories like “config”, “rules”, and parts of “model” handle parsing configuration files, alerting rules, recording rules, and integration with the Prometheus configuration and rule ecosystem.
Relationships and Data Flows
• Data flows from the scraping/discovery modules (collecting metrics) into the TSDB storage engine, where the WAL is managed and blocks of time series data are persisted.
• The query engine (PromQL parser) retrieves data from TSDB in response to API or web UI queries.
• Migration tools (prompptool) can be viewed as side utilities that are involved only during the transition or conversion phases—these influence how stored WAL data is converted before a full deployment.
• The API/UI component is exposed directly to external users via HTTP endpoints (and may also integrate with Prometheus Operator setups – as indicated by provided Kubernetes YAML examples).
• All these modules are integrated into a single monolithic binary (or set of binaries) while still maintaining a clear separation of concerns through the directory structure and “package” boundaries.
Architectural Patterns and Design Principles
• Modular and layered architecture: Even though the project is monolithic, it is logically divided into well-defined layers (ingestion, storage, query, and API/UI) with clear interfaces.
• Compatibility and drop-in replacement: The architecture uses the same interfaces (configuration, query API, storage block formats) as Prometheus, emphasizing backward compatibility.
• Performance optimization: Specific modules (the C++ parts in “pp” and sections in WAL and time series encoding/decoding) are optimized for lower memory usage and higher performance.
• Separation of utility functions: Conversion tools and secondary capabilities (such as rule evaluation and discovery) are kept in their own directories, ensuring clarity and maintainability.
Relevant Technologies, Frameworks, and Libraries
• Go: Most of the core components, such as the query engine, web server, and scraping logic are implemented in Go.
• C++ Optimizations: The “pp” directory indicates parts of the project written in C++ for optimized block and WAL handling.
• PromQL: The promql parsing and evaluation engine is central to executing queries.
• Docker, GitHub Workflows, and Kubernetes manifests are used for containerized deployment and CI/CD.
• Additional tools include Notifiers, Service Discovery modules, and third-party integrations (view “discovery”, “notifier”, “web”).
Elements
– Ingestion/Scraper & Discovery Module
– TSDB Storage & WAL Management (including the C++ optimized components)
– Query Engine & PromQL Parser
– API/Web UI layer
– Command Line Tools (prompptool/promtool)
– Data flow from ingestion into TSDB storage
– Query requests from the API/Web UI to the TSDB via the query engine
– Utility flow for WAL conversion between the conversion tool and the TSDB
– Core production components (scraping, TSDB, API)
– Auxiliary or migration components (conversion utilities)
– Infrastructure and external integrations (Prometheus Operator, Docker/Kubernetes)
• Component with key technologies (e.g., “Go-based query engine”, “C++ optimized storage module”, “HTTP API / Web UI”).