How thesis-dime/general-pipeline works
Project Type and Purpose
• This “general-pipeline” repository is a modular, config-driven AI pipeline. It orchestrates two main phases—image/text generation via a diffusion engine and multi-metric evaluation—while providing a full Docker-based deployment and observability stack.
Top-Level Components
a. Orchestrator
• main.py: Entry point that reads generation/evaluation configs and invokes each stage.
• compose.yml & install.sh: Define and bootstrap Docker Compose services.
b. Generation Service
• Submodule: DiffusionEngine
• Config: conf/generation/sd.yaml
c. Evaluation Services
• Submodules:
– EvalMuseMini (neural evaluation)
– fundamental_metrics (clip, fid, inception)
– QualiClip (semantic similarity)
• Configs under conf/evaluation/*
d. Monitoring & Observability
• Prometheus (monitoring/prometheus)
• Alertmanager (monitoring/alertmanager)
• Grafana + dashboards (monitoring/grafana/…)
• Caddy reverse proxy (monitoring/caddy)
e. Configuration Layer
• All YAMLs under conf/ define inputs to orchestrator and services.
f. Excluded Component
Architectural Patterns & Design Principles
• Modularity: Each pipeline stage lives in its own submodule and container.
• Config-driven Execution: No hard-coded parameters; runtime behavior controlled via YAML.
• Containerization: Docker Compose orchestrates all services—including core pipeline and monitoring.
• Observability: Prometheus-Grafana-Alertmanager stack for metrics and alerts.
Component Relationships & Data Flows
a. Orchestrator → Generation Service
• Reads sd.yaml → spins up DiffusionEngine container → generates artifacts.
b. Generation Service → Orchestrator
• Returns output locations or pushes to shared volume.
c. Orchestrator → Evaluation Services
• Sequentially invokes EvalMuseMini and fundamental_metrics containers, passing generated artifacts.
d. Evaluation Services → Orchestrator or Storage
• Emits metrics (JSON/CSV) back to orchestrator or to a results store.
e. All Containers → Prometheus
• Expose /metrics endpoints → scraped by Prometheus.
f. Prometheus → Alertmanager & Grafana
• Alerts triggered based on alert.rules → managed by Alertmanager; dashboards provisioned in Grafana.