# mahendrapaipuri/ceems architecture

> Architecture diagram of the GitHub repository mahendrapaipuri/ceems, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahendrapaipuri/ceems
- Repository: https://github.com/mahendrapaipuri/ceems

## Overview

##### Project type and purpose
- **Type:** A modular, backend‐focused “stack” composed of multiple Go microservices plus CLI tools, packaged for OS services or containers, without a built‐in UI.
- **Purpose:** Collect energy, performance, I/O and network metrics for “compute units” (VMs, containers, batch jobs, pods), expose them to Prometheus, aggregate and store them in a DB, provide a REST API for metadata and aggregated stats, enforce access control on time series storage, and supply CLI tools for ad‐hoc reporting.

##### File structure
Top‐level view:
- **cmd/** – multiple binaries (ceems_exporter, ceems_api_server, ceems_lb (load-balancer), redfish_proxy, cacct CLI, ceems_tool)
- **pkg/** – reusable libraries by concern:
– **collector/** (eBPF, IPMI, Redfish, cgroups)
– **api/** (HTTP server, DB migrations, models, resource‐manager integrations for SLURM, OpenStack, Kubernetes)
– **lb/** (TSDB load balancing & basic auth for Prometheus/Pyroscope)
– **emissions/** (carbon calculations)
– **grafana/** (HTTP discovery component integration)
– **tsdb/**, **sqlite3/**, **ipmi/**, **k8s/**, **lb/** subpackages
- **internal/** – low‐level helpers (os execution, security manager, structset)
- **build/**, **scripts/** – configuration templates, packaging, CI/CD support
- **etc/** – Docker, systemd, container and monitoring configs
- **website/** – documentation site (Docusaurus)

##### Key insights from README
- Four primary services: metrics exporter, REST API server, TSDB load‐balancer, optional Redfish proxy
- Relies on external systems: Prometheus (metrics scraping & TSDB), Grafana (visualization + HTTP discovery), Pyroscope (profiles)
- Local aggregated DB (SQLite with Litestream) for long‐term retention and REST API queries
- CLI tools for ad‐hoc reports and config generation

##### Be thorough but modular
- Binary, each library layer only if it informs a service boundary or integration.
- Don’t omit the aggregated DB behind the API server, nor the HTTP discovery piece in Grafana.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph: Node Agents
    subgraph "Node Host"
        direction TB
        Exporter["ceems_exporter"]:::service
        RedfishProxy["redfish_proxy (optional)"]:::service
        APIServer["ceems_api_server"]:::service
        SQLiteDB["SQLite (Litestream)"]:::database
        CLI_CACCT["cacct CLI"]:::cli
        CLI_TOOL["ceems_tool CLI"]:::cli
    end

    %% Subgraph: Resource Managers
    subgraph "Compute Units & Resource Managers" 
        direction TB
        SLURM["SLURM Adapter"]:::external
        OpenStack["OpenStack Adapter"]:::external
        K8S["Kubernetes Adapter"]:::external
    end

    %% Subgraph: Monitoring Stack
    subgraph "Monitoring & Storage"
        direction TB
        Prometheus["Prometheus"]:::external
        TSDBLB["TSDB Load Balancer"]:::service
        TSDB["TSDB Backend (Prom TSDB / Pyroscope)"]:::database
    end

    Grafana["Grafana + HTTP Discovery"]:::external

    %% Relationships
    SLURM -->|gRPC/REST| Exporter
    OpenStack -->|gRPC/REST| Exporter
    K8S -->|gRPC/REST| Exporter

    Exporter -->|HTTP scrape /metrics| Prometheus
    Exporter -->|BMC calls| RedfishProxy
    RedfishProxy -->|HTTP proxy| Exporter

    Prometheus -->|remote_write| TSDBLB
    Prometheus -->|remote_read| TSDBLB
    TSDBLB -->|write/read| TSDB

    Grafana -->|HTTP discovery| TSDBLB
    CLI_CACCT -->|HTTP read| TSDBLB

    Exporter -->|Exporter metrics| Prometheus

    APIServer -->|SQL CRUD| SQLiteDB
    CLI_CACCT -->|REST calls| APIServer
    CLI_TOOL -->|config gen| Exporter
    CLI_TOOL -->|config gen| APIServer
    CLI_TOOL -->|config gen| TSDBLB
    CLI_TOOL -->|config gen| RedfishProxy

    Grafana -->|uses| Grafana
    Grafana -.->|HTTP discovery| Grafana

    %% Click Events
    click SLURM "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/api/resource/slurm"
    click OpenStack "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/api/resource/openstack"
    click K8S "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/k8s"
    click Exporter "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/ceems_exporter/main.go"
    click Exporter "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/collector"
    click TSDBLB "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/ceems_lb/main.go"
    click TSDBLB "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/lb"
    click APIServer "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/ceems_api_server/main.go"
    click APIServer "https://github.com/mahendrapaipuri/ceems/tree/main/pkg/api"
    click SQLiteDB "https://github.com/mahendrapaipuri/ceems/blob/main/pkg/sqlite3/sqlite3.go"
    click SQLiteDB "https://github.com/mahendrapaipuri/ceems/blob/main/etc/litestream/config.yml"
    click CLI_CACCT "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/cacct/main.go"
    click CLI_TOOL "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/ceems_tool/main.go"
    click Grafana "https://github.com/mahendrapaipuri/ceems/blob/main/pkg/grafana/grafana.go"
    click Grafana "https://github.com/mahendrapaipuri/ceems/blob/main/etc/grafana/provisioning/datasources/ceems.yml"
    click Grafana "https://github.com/mahendrapaipuri/ceems/tree/main/thirdparty/grafana"
    click RedfishProxy "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/redfish_proxy/main.go"
    click RedfishProxy "https://github.com/mahendrapaipuri/ceems/blob/main/cmd/redfish_proxy/server.go"
    click RedfishProxy "https://github.com/mahendrapaipuri/ceems/blob/main/pkg/collector/redfish.go"

    %% Styles
    classDef service fill:#D0E8FF,stroke:#0366d6,stroke-width:1px;
    classDef database fill:#DFFFE0,stroke:#228B22,stroke-width:1px;
    classDef external fill:none,stroke:#888,stroke-dasharray: 5 5;
    classDef cli fill:#FFF5CC,stroke:#DAA520,stroke-width:1px;
```
