How populationgenomics/cpg-flow works
Main Components
• Configuration Component:
• Config files in TOML format (e.g., the example config file in the README and defaults.toml in src/cpg_flow).
• Environment variables (e.g. CPG_CONFIG_PATH) used to load configuration settings.
• Workflow Engine (Core Framework):
• The “workflow” module (src/cpg_flow/workflow.py) which handles running the pipeline and generating the DAG.
• Stage definitions (each stage located in src/cpg_flow, along with the “@stage” decorators that define dependencies).
• Job definitions (as seen in the jobs.py examples) that are queued via Hail Batch.
• DAG (Directed Acyclic Graph) Construction:
• The logic that represents how stages (e.g., GeneratePrimes, CumulativeCalc, FilterEvens, BuildAPrimePyramid) are chained via required_stages.
• The plotting mechanism for the DAG (triggered by the “show_workflow” flag) that uses tools like networkx and plotly.
• External and Infrastructure Components:
• Hail Batch as the backend execution engine (where jobs are scheduled and executed).
• Google Cloud (GCP) resources – storage (for inputs/outputs), Docker images pulled from GCP Container Registry.
• The analysis-runner component which is responsible for triggering the driver job on Hail Batch.
• Developer and CI/CD Components:
• Dockerfile and related GitHub Actions workflows for building/deploying the Docker image.
• Testing directories and tools (notably the tests/ directory for unit tests).
• Dependency management via uv and pre-commit configuration.
Relationships and Interactions
• That the configuration file feeds into the entrypoint (for example, main.py or workflow.py) to set up the pipeline.
• Workflow engine reading configuration, loading the pipeline definition (list of stages), and constructing a DAG.
• That each stage’s “expected_outputs” and “queue_jobs” functions integrate with Hail Batch jobs.
• Demonstrate that analysis-runner invokes the driver image which then triggers the workflow, and that job outputs (including the DAG plot) are logged and stored in GCP.
• Separate internal modules (stages, jobs, workflow management) from external services (Hail Batch, cloud storage, Docker registry).
Architectural Patterns and Design Principles
• Modularity – emphasize that pipeline stages are decoupled and implemented in separate files/directories (src/cpg_flow/targets, src/cpg_flow/stage.py, etc.).
• DAG-based workflow orchestration – show how stages depend on one another via backward discovery using the “required_stages” decorator.
• Extensibility – the system is designed so that new stages and jobs can be added without major rework.
• Loose Coupling with External Services – note that the framework delegates actual processing to Hail Batch jobs and integrates with cloud services for storage and execution.
Incorporate Relevant Technologies, Frameworks, and Tools
• Python as the primary language.
• Hail Batch for job orchestration.
• uv for dependency management.
• Graph libraries (networkx, plotly) to generate and view DAG plots.
• Docker for containerization, supported by deployment workflows on GitHub Actions.
• Pre-commit hooks and linting tools to enforce code quality.
– Configuration Manager (Config Files & Environment Variables)
– Workflow Manager (DAG Generator, Stage Scheduler)
– Job Scheduler (Integration with Hail Batch)
– External Services (GCP Storage, Hail Batch, Docker Registry)
– Rectangles for internal modules/components.
• Annotations where useful (for example, labeling how “backward discovery” works in the DAG and noting the explicit dependency chain).