# populationgenomics/cpg-flow architecture

> Architecture diagram of the GitHub repository populationgenomics/cpg-flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/populationgenomics/cpg-flow
- Repository: https://github.com/populationgenomics/cpg-flow

## Overview

##### Main Components
- Configuration Component:
- Config files in TOML format (e.g., the example config file in the README and defaults.toml in src/cpg_flow).
- Environment variables (e.g. CPG_CONFIG_PATH) used to load configuration settings.
- Workflow Engine (Core Framework):
- The “workflow” module (src/cpg_flow/workflow.py) which handles running the pipeline and generating the DAG.
- Stage definitions (each stage located in src/cpg_flow, along with the “@stage” decorators that define dependencies).
- Job definitions (as seen in the jobs.py examples) that are queued via Hail Batch.
- DAG (Directed Acyclic Graph) Construction:
- The logic that represents how stages (e.g., GeneratePrimes, CumulativeCalc, FilterEvens, BuildAPrimePyramid) are chained via required_stages.
- The plotting mechanism for the DAG (triggered by the “show_workflow” flag) that uses tools like networkx and plotly.
- External and Infrastructure Components:
- Hail Batch as the backend execution engine (where jobs are scheduled and executed).
- Google Cloud (GCP) resources – storage (for inputs/outputs), Docker images pulled from GCP Container Registry.
- The analysis-runner component which is responsible for triggering the driver job on Hail Batch.
- Developer and CI/CD Components:
- Dockerfile and related GitHub Actions workflows for building/deploying the Docker image.
- Testing directories and tools (notably the tests/ directory for unit tests).
- Dependency management via uv and pre-commit configuration.

##### Relationships and Interactions
- That the configuration file feeds into the entrypoint (for example, main.py or workflow.py) to set up the pipeline.
- Workflow engine reading configuration, loading the pipeline definition (list of stages), and constructing a DAG.
- That each stage’s “expected_outputs” and “queue_jobs” functions integrate with Hail Batch jobs.
- Demonstrate that analysis-runner invokes the driver image which then triggers the workflow, and that job outputs (including the DAG plot) are logged and stored in GCP.
- Separate internal modules (stages, jobs, workflow management) from external services (Hail Batch, cloud storage, Docker registry).

##### Architectural Patterns and Design Principles
- Modularity – emphasize that pipeline stages are decoupled and implemented in separate files/directories (src/cpg_flow/targets, src/cpg_flow/stage.py, etc.).
- DAG-based workflow orchestration – show how stages depend on one another via backward discovery using the “required_stages” decorator.
- Extensibility – the system is designed so that new stages and jobs can be added without major rework.
- Loose Coupling with External Services – note that the framework delegates actual processing to Hail Batch jobs and integrates with cloud services for storage and execution.

##### Incorporate Relevant Technologies, Frameworks, and Tools
- Python as the primary language.
- Hail Batch for job orchestration.
- uv for dependency management.
- Graph libraries (networkx, plotly) to generate and view DAG plots.
- Docker for containerization, supported by deployment workflows on GitHub Actions.
- Pre-commit hooks and linting tools to enforce code quality.

– Configuration Manager (Config Files & Environment Variables)
– Workflow Manager (DAG Generator, Stage Scheduler)
– Job Scheduler (Integration with Hail Batch)
– External Services (GCP Storage, Hail Batch, Docker Registry)
– Rectangles for internal modules/components.
- Annotations where useful (for example, labeling how “backward discovery” works in the DAG and noting the explicit dependency chain).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Manager
    subgraph "Configuration Manager"
        A1["README Config (Config Example)"]:::internal
        A2["Defaults TOML (Config Defaults)"]:::internal
        A3["Environment Variables (CPG_CONFIG_PATH)"]:::internal
    end

    %% Workflow Engine with Stage Modules
    subgraph "Workflow Engine"
        B1["Workflow Module (DAG Generator & Stage Scheduler)"]:::internal
        subgraph "Stage Modules"
            B2["Stage Core (Dependency Management)"]:::internal
            S1["Cohort Stage"]:::internal
            S2["Dataset Stage"]:::internal
        end
    end

    %% DAG Visualization
    subgraph "DAG Visualization"
        C1["DAG Plotting (networkx/plotly)"]:::internal
    end

    %% Job Scheduling
    subgraph "Job Scheduling"
        D1["Hail Batch Job Scheduler"]:::internal
    end

    %% External and Infrastructure Components
    subgraph "External Services"
        E1["Hail Batch Execution"]:::external
        E2["GCP Storage"]:::external
        E3["Docker Registry"]:::external
        E4["Analysis Runner (Driver Job)"]:::external
    end

    %% Developer & CI/CD Components
    subgraph "Developer & CI/CD"
        F1["CI/CD Workflows (.github/workflows)"]:::dev
        F2["Dockerfile (Container Build)"]:::dev
        F3["Pre-commit Hooks (.pre-commit-config.yaml)"]:::dev
        F4["Testing Suite (tests/)"]:::dev
        F5["Dependency Management (uv.lock & pyproject.toml)"]:::dev
    end

    %% Relationships and Data Flow
    A1 -->|"loads"| B1
    A2 -->|"loads"| B1
    A3 -->|"configures"| B1

    B1 -->|"loads stages"| B2
    B2 -->|"defines dependency chain"| S1
    B2 -->|"defines dependency chain"| S2
    B2 -->|"builds DAG"| C1
    B2 -->|"queues jobs"| D1

    D1 -->|"executes jobs"| E1
    E1 -->|"stores outputs"| E2
    E4 -->|"triggers"| D1

    F1 -->|"builds and deploys"| F2
    F2 -->|"pushes image"| E3

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef external fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef dev fill:#fff3cd,stroke:#856404,stroke-width:2px;

    %% Click Events for Component Mapping
    click A1 "https://github.com/populationgenomics/cpg-flow/blob/main/README.md"
    click A2 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/defaults.toml"
    click B1 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/workflow.py"
    click B2 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/stage.py"
    click S1 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/targets/cohort.py"
    click S2 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/targets/dataset.py"
    click C1 "https://github.com/populationgenomics/cpg-flow/blob/main/src/cpg_flow/show_workflow/graph.py"
    click F1 "https://github.com/populationgenomics/cpg-flow/tree/main/.github/workflows/"
    click F2 "https://github.com/populationgenomics/cpg-flow/tree/main/Dockerfile"
    click F3 "https://github.com/populationgenomics/cpg-flow/blob/main/.pre-commit-config.yaml"
    click F4 "https://github.com/populationgenomics/cpg-flow/tree/main/tests/"
    click F5 "https://github.com/populationgenomics/cpg-flow/blob/main/uv.lock"
```
