# cbg-ethz/v-pipe architecture

> Architecture diagram of the GitHub repository cbg-ethz/v-pipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cbg-ethz/v-pipe
- Repository: https://github.com/cbg-ethz/v-pipe

## Overview

##### Major components
- Input & Configuration:
– Input samples: Notice the “samples/” directory convention (mentioned in the README and documented under “config/README.md”) where the raw sequencing data are organized in a two-level hierarchy with “raw_data” sub-folders.
– Configuration files: The “config” directory contains the main configuration (YAML/HTML) along with virus-specific base configuration files (e.g., hiv, sars-cov-2) that determine which parameters to use.
– Utility scripts: Under “utils/” there are mass-importers and quick installer scripts.
- The Workflow Engine:
– Core Snakemake engine: The “workflow/” directory is the heart of the pipeline. It includes the main Snakefile, environment definitions (conda YAML files under “workflow/envs”), rule definitions (in “workflow/rules”), and schema validations.
– Auxiliary workflows: Also note that in “resources/auxiliary_workflows” there are benchmarking and multi-setup workflows that complement the main pipeline.
- Processing Modules:
– Preprocessing and Quality Assessment: Tools like FastQC, PRINSEQ, and preprocessing rules defined in “workflow/rules/preprocessing.smk”.
– Read Alignment and Reference Handling: Rules for alignment (using ngshmmalign, bwa, minimap, etc.) and consensus building; reference data provided in “resources/” (with virus-specific fasta files, gff/gff3, and primers).
– Variant Calling & Haplotype Reconstruction: SNV calling (likely using ShoRAH, LoFreq, etc.), local and global haplotype assembly (e.g., using Haploclique, SAVAGE) – rules are organized in individual Snakemake files.
– Postprocessing & Visualization: Rules and scripts for generating metrics, statistics, and visualizations in “workflow/scripts” and the “docs/” directory.
- Deployment/Execution & CI Infrastructure:
– Installation/Docker: The README details different deployment methods (quick install, Docker container, Snakedeploy). The Dockerfile at the root and GitHub workflows in “.github/workflows” are also present.
– Testing and Validation: The “tests/” folder contains sample data, test scripts, and expected outputs for regression tests.
– Continuous Integration: There are GitHub workflows (e.g., mega-linter.yml, test-docker.yaml, run_regression_tests.yaml) that enforce code quality, linting, and regression testing.

##### Relationships and Interactions
– User Input → Configuration:
- The user organizes input files (via the “samples/” directory) and customizes the working configuration (config.yaml) which sets virus-specific parameters.
– Configuration → Workflow Engine:
- Snakemake loads the configuration and selects task rules accordingly.
– Workflow Engine → Processing Modules:
- The central Snakefile orchestrates the execution of preprocessing, alignment, variant calling, haplotype reconstruction and visualization. Each module calls specific external computational tools (like FastQC, ngshmmalign, and others) using dedicated conda environments.
– Auxiliary and CI Processes:
- Auxiliary workflows (benchmarking, multi_setup) are used for performance evaluation and further method definitions stored under “resources/auxiliary_workflows”.
- The GitHub workflows automate testing, container deploys, and linting.
– Output Production:
- Final outputs (consensus sequences, variant call files like VCFs, visualization files, quality assurance metrics) are produced and saved in designated output directories as defined in the configuration.

##### Architectural Patterns & Design Principles
– Modular & Extensible: The pipeline splits into many small and independent rules (Snakemake rules) that can be easily maintained or extended.
– Environment Isolation: Each rule is executed in its own conda environment (or Docker container) ensuring reproducibility.
– Separation of Concern & Config-Driven: The use of dedicated configuration files (per virus and per environment) separates execution logic from the parameters and sample-specific details.
– CI/CD Integration: GitHub Actions ensure that testing and deployment are automated.

##### Technologies & Dependencies
– Snakemake as the workflow manager is central to the orchestration.
– Conda (and mamba) environments are used for dependency management.
– Docker is implemented as an alternative deployment mode.
– Various external tools (FastQC, ngshmmalign, bwa, MAFFT, etc.) are managed by Snakemake rules.
– Standard languages: Python and shell scripts are used in “workflow/scripts” and utils.

##### Details Specific to V‐pipe
– Emphasize virus-specific configuration (with base configs for HIV, SARS-CoV-2, etc.) and note where reference files and primers are stored (in the “resources/” subdirectory).
– Represent that the pipeline can be deployed in multiple ways (locally, via Docker, or using Snakedeploy) as described in the README.
– Detail that there is robust testing and CI infrastructure (with GitHub workflows, regression tests from “tests/” folder).

– The User taking input (sample directory and configuration files)
– The central Snakemake engine (with its main Snakefile, rules and conda environments)
– Sequentially arranged processing modules (preprocessing, alignment, variant calling, haplotype reconstruction, visualization)
– Auxiliary workflows for benchmarking and testing
– Deployment paths (quick install script, Docker, Snakedeploy)
– External dependencies linked to each processing module

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Inputs & Configuration
    subgraph "User Inputs & Configuration"
        A1["Samples"]:::config
        A2["Configuration Files"]:::config
        A3["Utility Scripts"]:::utility
    end

    %% Workflow Engine (Snakemake)
    subgraph "Workflow Engine (Snakemake)"
        B1["Main Snakefile"]:::engine
        B2["Rules"]:::engine
        B3["Conda Environments"]:::engine
        B4["Schema Validations"]:::engine
    end

    %% Processing Modules
    subgraph "Processing Modules"
        C1["Preprocessing & QC"]:::module
        C2["Alignment & Reference Handling"]:::module
        C3["Variant Calling & Haplotype Reconstruction"]:::module
        C4["Postprocessing & Visualization"]:::module
    end

    %% Auxiliary Workflows & CI
    subgraph "Auxiliary Workflows & CI"
        D1["Auxiliary Workflows & Benchmarking"]:::aux
        D2["CI & Testing"]:::aux
    end

    %% Deployment & Execution
    subgraph "Deployment & Execution"
        E1["Docker Deployment"]:::deploy
        E2["Quick Install"]:::deploy
    end

    %% External Tools
    F1["External Tools (FastQC, ngshmmalign, etc.)"]:::tool

    %% Final Outputs
    O1["Final Outputs (Consensus, VCFs, Visualizations)"]:::output

    %% Data Flow
    A1 -->|"data"| B1
    A2 -->|"config"| B1
    A3 -->|"scripts"| B1

    B1 -->|"orchestrates"| C1
    B1 -->|"orchestrates"| C2
    B1 -->|"orchestrates"| C3
    B1 -->|"orchestrates"| C4

    B3 -->|"provides environment"| C1
    B3 -->|"provides environment"| C2
    B3 -->|"provides environment"| C3
    B3 -->|"provides environment"| C4

    B1 -->|"triggers"| D1
    B1 -->|"triggers"| D2

    E1 -->|"deploys"| B1
    E2 -->|"deploys"| B1

    C1 -->|"generates"| O1
    C2 -->|"generates"| O1
    C3 -->|"generates"| O1
    C4 -->|"generates"| O1

    F1 ---|"support"| C1
    F1 ---|"support"| C2
    F1 ---|"support"| C3
    F1 ---|"support"| C4

    %% Click Events
    click A1 "https://github.com/cbg-ethz/v-pipe/tree/master/docs/example_HIV_data/samples"
    click A2 "https://github.com/cbg-ethz/v-pipe/tree/master/config/"
    click A3 "https://github.com/cbg-ethz/v-pipe/blob/master/utils/quick_install.sh"
    click B1 "https://github.com/cbg-ethz/v-pipe/tree/master/workflow/Snakefile"
    click B2 "https://github.com/cbg-ethz/v-pipe/tree/master/workflow/rules/"
    click B3 "https://github.com/cbg-ethz/v-pipe/tree/master/workflow/envs/"
    click B4 "https://github.com/cbg-ethz/v-pipe/blob/master/workflow/schemas/config_schema.json"
    click C1 "https://github.com/cbg-ethz/v-pipe/blob/master/workflow/rules/quality_assurance.smk"
    click C2 "https://github.com/cbg-ethz/v-pipe/blob/master/workflow/rules/align.smk"
    click C3 "https://github.com/cbg-ethz/v-pipe/blob/master/workflow/rules/snv.smk"
    click C4 "https://github.com/cbg-ethz/v-pipe/blob/master/workflow/rules/visualization.smk"
    click D1 "https://github.com/cbg-ethz/v-pipe/tree/master/resources/auxiliary_workflows/"
    click D2 "https://github.com/cbg-ethz/v-pipe/tree/master/.github/workflows/"
    click E1 "https://github.com/cbg-ethz/v-pipe/tree/master/Dockerfile"
    click E2 "https://github.com/cbg-ethz/v-pipe/blob/master/utils/quick_install.sh"

    %% Styles
    classDef config fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef engine fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef module fill:#ffff99,stroke:#000,stroke-width:2px;
    classDef aux fill:#ffcc99,stroke:#000,stroke-width:2px;
    classDef deploy fill:#d3d3d3,stroke:#000,stroke-width:2px;
    classDef tool fill:#f5b7b1,stroke:#000,stroke-width:2px;
    classDef output fill:#e6e6fa,stroke:#000,stroke-width:2px;
    classDef utility fill:#ffc0cb,stroke:#000,stroke-width:2px;
```
