# thesis-dime/general-pipeline architecture

> Architecture diagram of the GitHub repository thesis-dime/general-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thesis-dime/general-pipeline
- Repository: https://github.com/thesis-dime/general-pipeline

## Overview

##### Project Type and Purpose
- This “general-pipeline” repository is a modular, config-driven AI pipeline. It orchestrates two main phases—image/text generation via a diffusion engine and multi-metric evaluation—while providing a full Docker-based deployment and observability stack.

##### Top-Level Components
a. Orchestrator
- main.py: Entry point that reads generation/evaluation configs and invokes each stage.
- compose.yml & install.sh: Define and bootstrap Docker Compose services.
b. Generation Service
- Submodule: DiffusionEngine
- Config: conf/generation/sd.yaml
c. Evaluation Services
- Submodules:
– EvalMuseMini (neural evaluation)
– fundamental_metrics (clip, fid, inception)
– QualiClip (semantic similarity)
- Configs under conf/evaluation/*
d. Monitoring & Observability
- Prometheus (monitoring/prometheus)
- Alertmanager (monitoring/alertmanager)
- Grafana + dashboards (monitoring/grafana/…)
- Caddy reverse proxy (monitoring/caddy)
e. Configuration Layer
- All YAMLs under conf/ define inputs to orchestrator and services.
f. Excluded Component

##### Architectural Patterns & Design Principles
- Modularity: Each pipeline stage lives in its own submodule and container.
- Config-driven Execution: No hard-coded parameters; runtime behavior controlled via YAML.
- Containerization: Docker Compose orchestrates all services—including core pipeline and monitoring.
- Observability: Prometheus-Grafana-Alertmanager stack for metrics and alerts.

##### Component Relationships & Data Flows
a. Orchestrator → Generation Service
- Reads sd.yaml → spins up DiffusionEngine container → generates artifacts.
b. Generation Service → Orchestrator
- Returns output locations or pushes to shared volume.
c. Orchestrator → Evaluation Services
- Sequentially invokes EvalMuseMini and fundamental_metrics containers, passing generated artifacts.
d. Evaluation Services → Orchestrator or Storage
- Emits metrics (JSON/CSV) back to orchestrator or to a results store.
e. All Containers → Prometheus
- Expose /metrics endpoints → scraped by Prometheus.
f. Prometheus → Alertmanager & Grafana
- Alerts triggered based on alert.rules → managed by Alertmanager; dashboards provisioned in Grafana.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        confConfig["conf/config.yaml"]:::config
        click confConfig "https://github.com/thesis-dime/general-pipeline/blob/main/conf/config.yaml"
        confGenDir["conf/generation"]:::config
        click confGenDir "https://github.com/thesis-dime/general-pipeline/tree/main/conf/generation"
        confEvalDir["conf/evaluation"]:::config
        click confEvalDir "https://github.com/thesis-dime/general-pipeline/tree/main/conf/evaluation"
    end

    %% Core Pipeline
    subgraph "Core Pipeline"
        direction TB
        compose["Docker Compose & Bootstrap"]:::core
        click compose "https://github.com/thesis-dime/general-pipeline/blob/main/compose.yml"
        install["install.sh"]:::core
        click install "https://github.com/thesis-dime/general-pipeline/blob/main/install.sh"
        orchestrator["Orchestrator (main.py)"]:::core
        click orchestrator "https://github.com/thesis-dime/general-pipeline/blob/main/main.py"

        compose --> orchestrator
        orchestrator --> confConfig
        orchestrator --> confGenDir
        orchestrator --> confEvalDir

        subgraph "Submodules"
            direction TB
            genService["Generation Service"]:::core
            subDiff["DiffusionEngine"]:::external
            click subDiff "https://github.com/thesis-dime/general-pipeline/tree/main/submodules/DiffusionEngine"
            sdConfig["sd.yaml"]:::config
            click sdConfig "https://github.com/thesis-dime/general-pipeline/blob/main/conf/generation/sd.yaml"

            evalMuse["EvalMuseMini"]:::external
            click evalMuse "https://github.com/thesis-dime/general-pipeline/tree/main/submodules/EvalMuseMini"
            evalMuseConf["eval_muse.yaml"]:::config
            click evalMuseConf "https://github.com/thesis-dime/general-pipeline/blob/main/conf/evaluation/eval_muse/eval_muse.yaml"

            fund["Fundamental Metrics"]:::external
            click fund "https://github.com/thesis-dime/general-pipeline/tree/main/submodules/fundamental_metrics"
            fundConf["fundamental_metrics.yaml"]:::config
            click fundConf "https://github.com/thesis-dime/general-pipeline/blob/main/conf/evaluation/fundamental_metrics/fundamental_metrics.yaml"
            clipScore["clip_score.yaml"]:::config
            click clipScore "https://github.com/thesis-dime/general-pipeline/blob/main/conf/evaluation/fundamental_metrics/metrics/clip_score.yaml"
            fidScore["fid_score.yaml"]:::config
            click fidScore "https://github.com/thesis-dime/general-pipeline/blob/main/conf/evaluation/fundamental_metrics/metrics/fid_score.yaml"
            inceptionScore["inception_score.yaml"]:::config
            click inceptionScore "https://github.com/thesis-dime/general-pipeline/blob/main/conf/evaluation/fundamental_metrics/metrics/inception_score.yaml"

            quali["QualiClip"]:::external
            click quali "https://github.com/thesis-dime/general-pipeline/tree/main/submodules/QualiClip"

            orchestrator --> genService
            genService --> subDiff
            genService --> sdConfig
            genService --> orchestrator

            orchestrator --> evalMuse
            evalMuse --> evalMuseConf
            evalMuse --> orchestrator

            orchestrator --> fund
            fund --> fundConf
            fund --> clipScore
            fund --> fidScore
            fund --> inceptionScore
            fund --> orchestrator

            orchestrator --> quali
            quali --> orchestrator
        end
    end

    %% Monitoring & Observability
    subgraph "Monitoring & Observability"
        direction TB
        prom["Prometheus"]:::monitor
        click prom "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/prometheus/prometheus.yml"
        click promRules "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/prometheus/alert.rules"
        alertm["Alertmanager"]:::monitor
        click alertm "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/alertmanager/config.yml"
        graf["Grafana"]:::monitor
        click grafDash "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/grafana/provisioning/dashboards/*.json"
        click grafDashY "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/grafana/provisioning/dashboards/dashboard.yml"
        click grafDS "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/grafana/provisioning/datasources/datasource.yml"
        caddy["Caddy Proxy"]:::monitor
        click caddy "https://github.com/thesis-dime/general-pipeline/tree/main/monitoring/caddy/Caddyfile"
        monComp["Monitoring Compose"]:::monitor
        click monComp "https://github.com/thesis-dime/general-pipeline/blob/main/monitoring/compose.yml"

        prom --> alertm
        prom --> graf

        orchestrator -.->|"/metrics"| prom
        genService -.->|"/metrics"| prom
        evalMuse -.->|"/metrics"| prom
        fund -.->|"/metrics"| prom
        quali -.->|"/metrics"| prom
    end

    %% Styling
    classDef core fill:#AED6F1,stroke:#21618C;
    classDef config fill:#FAD7A0,stroke:#B9770E;
    classDef monitor fill:#ABEBC6,stroke:#1D8348;
    classDef external fill:#D5DBDB,stroke:#797D7F;
```
