# deckhouse/prompp architecture

> Architecture diagram of the GitHub repository deckhouse/prompp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deckhouse/prompp
- Repository: https://github.com/deckhouse/prompp

## Overview

The Deckhouse Prom++ project is a high-performance fork of Prometheus that retains full compatibility with its configuration, API endpoints, storage formats, and overall functionality. However, it introduces major optimizations by rewriting the most resource-intensive components (namely the in-memory block construction and WAL (Write-Ahead Log) management) in C++. This has the effect of drastically reducing memory consumption (up to 10x reduction) while still operating as a drop‐in replacement for Prometheus.

##### Main Components
- Ingestion & Scraping Module: Represented by directories such as “scrape”, “discovery”, and “notifier”. These modules handle receiving metric data from various targets, applying service discoveries (integrations with Kubernetes, AWS, etc.), and then forwarding or notifying as needed.
- WAL Management & TSDB Storage: A key part of Prom++ is its optimized time-series database (TSDB) and WAL subsystem. Look at directories such as “tsdb”, “storage”, and “pp” (which contains the C++ optimized components including WAL handling and block construction). This is the engine that writes, reads, compacts, and serves stored metrics.
- Query Engine & PromQL Parser: The “promql” directory (with its engine, parser, and testing files), together with parts of “model”, encapsulate the logic for parsing and executing PromQL queries.
- API, Web UI & HTTP Server: The “web” directory, along with parts of “cmd/prometheus”, support API endpoints, the HTTP UI, and federation features. This component serves queries and configuration endpoints.
- Migration & Conversion Tools: The “cmd/prompptool” and “cmd/promtool” directories contain command-line utilities that automate WAL conversion, enabling seamless migration between Prometheus’s original and Prom++ formats.
- Configuration and Rules Engine: Directories like “config”, “rules”, and parts of “model” handle parsing configuration files, alerting rules, recording rules, and integration with the Prometheus configuration and rule ecosystem.

##### Relationships and Data Flows
- Data flows from the scraping/discovery modules (collecting metrics) into the TSDB storage engine, where the WAL is managed and blocks of time series data are persisted.
- The query engine (PromQL parser) retrieves data from TSDB in response to API or web UI queries.
- Migration tools (prompptool) can be viewed as side utilities that are involved only during the transition or conversion phases—these influence how stored WAL data is converted before a full deployment.
- The API/UI component is exposed directly to external users via HTTP endpoints (and may also integrate with Prometheus Operator setups – as indicated by provided Kubernetes YAML examples).
- All these modules are integrated into a single monolithic binary (or set of binaries) while still maintaining a clear separation of concerns through the directory structure and “package” boundaries.

##### Architectural Patterns and Design Principles
- Modular and layered architecture: Even though the project is monolithic, it is logically divided into well-defined layers (ingestion, storage, query, and API/UI) with clear interfaces.
- Compatibility and drop-in replacement: The architecture uses the same interfaces (configuration, query API, storage block formats) as Prometheus, emphasizing backward compatibility.
- Performance optimization: Specific modules (the C++ parts in “pp” and sections in WAL and time series encoding/decoding) are optimized for lower memory usage and higher performance.
- Separation of utility functions: Conversion tools and secondary capabilities (such as rule evaluation and discovery) are kept in their own directories, ensuring clarity and maintainability.

##### Relevant Technologies, Frameworks, and Libraries
- Go: Most of the core components, such as the query engine, web server, and scraping logic are implemented in Go.
- C++ Optimizations: The “pp” directory indicates parts of the project written in C++ for optimized block and WAL handling.
- PromQL: The promql parsing and evaluation engine is central to executing queries.
- Docker, GitHub Workflows, and Kubernetes manifests are used for containerized deployment and CI/CD.
- Additional tools include Notifiers, Service Discovery modules, and third-party integrations (view “discovery”, “notifier”, “web”).

##### Elements
– Ingestion/Scraper & Discovery Module
– TSDB Storage & WAL Management (including the C++ optimized components)
– Query Engine & PromQL Parser
– API/Web UI layer
– Command Line Tools (prompptool/promtool)
– Data flow from ingestion into TSDB storage
– Query requests from the API/Web UI to the TSDB via the query engine
– Utility flow for WAL conversion between the conversion tool and the TSDB
– Core production components (scraping, TSDB, API)
– Auxiliary or migration components (conversion utilities)
– Infrastructure and external integrations (Prometheus Operator, Docker/Kubernetes)
- Component with key technologies (e.g., “Go-based query engine”, “C++ optimized storage module”, “HTTP API / Web UI”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    Targets["Targets & Exporters"]:::external
    Deployment["Deployment (Docker/Kubernetes)"]:::external

    %% Ingestion & Scraping Modules
    subgraph "Ingestion & Scraping Modules"
        Scrape["Scrape Module"]:::ingestion
        Discovery["Discovery Module"]:::ingestion
        Notifier["Notifier Module"]:::ingestion
    end

    %% TSDB Storage & WAL Management
    subgraph "TSDB Storage & WAL Management"
        TSDB["TSDB"]:::storage
        Storage["Storage Interface"]:::storage
        CPPOptimized["C++ Optimized (WAL & Block)"]:::storage
    end

    %% Query Engine & PromQL Parser
    subgraph "Query Engine & PromQL Parser"
        PromQL["PromQL Parser"]:::query
        Model["Data Model & Rules"]:::query
    end

    %% Configuration & Rules Engine
    subgraph "Configuration & Rules Engine"
        Config["Configuration"]:::config
        Rules["Rules Engine"]:::config
    end

    %% API, Web UI & HTTP Server
    subgraph "API & Web UI"
        Web["Web UI"]:::api
        HTTPServer["HTTP Server (cmd/prometheus)"]:::api
    end

    %% Migration & Conversion Tools
    subgraph "Migration & Conversion Tools"
        Prompptool["Prompptool"]:::migration
        Promtool["Promtool"]:::migration
    end

    %% Data Flow Connections
    Targets -->|"feeds"| Scrape
    Targets -->|"feeds"| Discovery
    Targets -->|"feeds"| Notifier

    %% Ingestion forwards data to Storage Engine
    Scrape -->|"ingests"| TSDB
    Discovery -->|"ingests"| TSDB
    Notifier -->|"alerts"| TSDB

    %% Storage Layer internal flow (Storage and C++ optimizations work in tandem)
    TSDB ---|"interfaces with"| Storage
    TSDB ---|"optimized by"| CPPOptimized

    %% Query Flow from Storage to Query Engine
    TSDB -->|"query data"| PromQL

    %% Configuration influences Model & Rules
    Config -->|"configures"| Model
    Rules -->|"defines"| Model

    %% Query Engine processing
    Model ---|"parses"| PromQL

    %% Query responses via API Layer
    PromQL -->|"serves queries"| HTTPServer
    PromQL -->|"serves queries"| Web

    %% Deployment integration
    Deployment -->|"accesses"| HTTPServer

    %% Migration Tools interface with Storage for conversion
    Prompptool ---|"converts WAL"| TSDB
    Promtool ---|"converts WAL"| TSDB

    %% Styling Classes
    classDef ingestion fill:#E3F2FD,stroke:#1E88E5,stroke-width:2px;
    classDef storage fill:#E8F5E9,stroke:#43A047,stroke-width:2px;
    classDef query fill:#FFF3E0,stroke:#FB8C00,stroke-width:2px;
    classDef api fill:#FFEBEE,stroke:#E53935,stroke-width:2px;
    classDef migration fill:#F3E5F5,stroke:#8E24AA,stroke-width:2px;
    classDef config fill:#ECEFF1,stroke:#37474F,stroke-width:2px;
    classDef external fill:#FFFDE7,stroke:#FBC02D,stroke-width:2px;

    %% Click Events for Ingestion & Scraping Modules
    click Scrape "https://github.com/deckhouse/prompp/tree/pp/scrape"
    click Discovery "https://github.com/deckhouse/prompp/tree/pp/discovery"
    click Notifier "https://github.com/deckhouse/prompp/tree/pp/notifier"

    %% Click Events for TSDB Storage & WAL Management
    click TSDB "https://github.com/deckhouse/prompp/tree/pp/tsdb"
    click Storage "https://github.com/deckhouse/prompp/tree/pp/storage"
    click CPPOptimized "https://github.com/deckhouse/prompp/tree/pp/pp"

    %% Click Events for Query Engine & PromQL Parser
    click PromQL "https://github.com/deckhouse/prompp/tree/pp/promql"
    click Model "https://github.com/deckhouse/prompp/tree/pp/model"

    %% Click Events for API, Web UI & HTTP Server
    click Web "https://github.com/deckhouse/prompp/tree/pp/web"
    click HTTPServer "https://github.com/deckhouse/prompp/tree/pp/cmd/prometheus"

    %% Click Events for Migration & Conversion Tools
    click Prompptool "https://github.com/deckhouse/prompp/tree/pp/cmd/prompptool"
    click Promtool "https://github.com/deckhouse/prompp/tree/pp/cmd/promtool"

    %% Click Events for Configuration & Rules Engine
    click Config "https://github.com/deckhouse/prompp/tree/pp/config"
    click Rules "https://github.com/deckhouse/prompp/tree/pp/rules"
```
