How mahendrapaipuri/ceems works
Project type and purpose
• Type: A modular, backend‐focused “stack” composed of multiple Go microservices plus CLI tools, packaged for OS services or containers, without a built‐in UI.
• Purpose: Collect energy, performance, I/O and network metrics for “compute units” (VMs, containers, batch jobs, pods), expose them to Prometheus, aggregate and store them in a DB, provide a REST API for metadata and aggregated stats, enforce access control on time series storage, and supply CLI tools for ad‐hoc reporting.
File structure
Top‐level view:
• cmd/ – multiple binaries (ceems_exporter, ceems_api_server, ceems_lb (load-balancer), redfish_proxy, cacct CLI, ceems_tool)
• pkg/ – reusable libraries by concern:
– collector/ (eBPF, IPMI, Redfish, cgroups)
– api/ (HTTP server, DB migrations, models, resource‐manager integrations for SLURM, OpenStack, Kubernetes)
– lb/ (TSDB load balancing & basic auth for Prometheus/Pyroscope)
– emissions/ (carbon calculations)
– grafana/ (HTTP discovery component integration)
– tsdb/, sqlite3/, ipmi/, k8s/, lb/ subpackages
• internal/ – low‐level helpers (os execution, security manager, structset)
• build/, scripts/ – configuration templates, packaging, CI/CD support
• etc/ – Docker, systemd, container and monitoring configs
• website/ – documentation site (Docusaurus)
Key insights from README
• Four primary services: metrics exporter, REST API server, TSDB load‐balancer, optional Redfish proxy
• Relies on external systems: Prometheus (metrics scraping & TSDB), Grafana (visualization + HTTP discovery), Pyroscope (profiles)
• Local aggregated DB (SQLite with Litestream) for long‐term retention and REST API queries
• CLI tools for ad‐hoc reports and config generation
Be thorough but modular
• Binary, each library layer only if it informs a service boundary or integration.
• Don’t omit the aggregated DB behind the API server, nor the HTTP discovery piece in Grafana.