How comcast/fishymetrics works
Project Type and Purpose
• This is a backend service – a Prometheus exporter named “fishymetrics.”
• It scrapes hardware management stats from bare-metal chassis via the Redfish API (and pluggable non-Redfish APIs) and exposes metrics over HTTP for Prometheus.
• It also provides build-info and runtime metrics endpoints, supports credential management via Vault or static config, and offers pluggable extensions for additional hardware models.
File Structure Analysis
Top-level layout:
• cmd/fishymetrics – the main CLI/app entry point and template handling
• exporter – HTTP handlers, core scraper logic, metrics construction, plus a nested moonshot exporter
• oem – model-specific Redfish wrappers per chassis type (generic + moonshot folder)
• plugins – extension points for non-Redfish or custom collectors (e.g., Nuova)
• common, config, pool, vault, logger, middleware – cross-cutting libraries for credentials, configuration, concurrency, secret retrieval, logging, and HTTP middleware
• helm – Kubernetes deployment charts
• Dockerfile/Makefile – build/test/deploy automation
• .github – CI/security workflows and issue templates
Key Architectural Insights from README
• Single HTTP service exposing /metrics, /scrape, /info endpoints
• Flags/env-var configuration for BMC creds, Vault integration, logging, collector filters
• Prometheus scrapes the /scrape path with URL params to target specific chassis models/addresses
• Concurrency handled via a worker pool for parallel Redfish calls
• Middleware for request logging and Prometheus internal metrics
Main Components
a. fishymetrics Server
– cmd/fishymetrics: application bootstrap, flag parsing, HTTP server start
b. HTTP Layer
– middleware (logging, muxprom)
– handlers (/metrics, /info, /scrape) in exporter/handlers.go
c. Credential Management
– static credentials or Vault integration (vault package + common/credentials)
d. Scraping Layer
– exporter core: orchestrates collectors, applies exclude filters
– pool: concurrency management for parallel Redfish/tasks
– oem modules: Redfish API clients per chassis model
– plugins: custom handlers for non-Redfish targets
e. Metrics Assembly
– exporter/metrics.go: builds Prometheus metric families
– Prometheus client library
f. External Dependencies
– Redfish-enabled BMCs (HTTP REST endpoints)
– Vault server (for AppRole credential retrieval)
– Prometheus (scrapes metrics)
– Vector or file sink (for structured logging)
Relationships and Data Flow
1. Prometheus → HTTP GET /scrape on fishymetrics (with query params)
2. HTTP middleware logs request + records internal metrics
3. Handler invokes credential loader (Vault or static)
4. Handler submits tasks to pool: for each collector (oem module or plugin)
5. Collector contacts BMC Redfish API (HTTPS) and returns raw JSON/stats
6. Aggregated data passed to exporter/metrics, converted into Prometheus metrics
7. /metrics endpoint serves exporter’s own runtime metrics
8. Logs emitted to stdout/file or Vector endpoint
Important Architectural Patterns and Principles
• Modular design: separate packages for concerns (config, logging, HTTP, scraping)
• Pluggable collectors via plugins/ and exporter/moonshot
• Concurrency via a worker pool to bound parallel Redfish calls
• Middleware for cross-cutting concerns (logging, instrumentation)
• Twelve-factor style config (flags + env vars)
• Containerized deployment with Docker + Helm