How cbg-ethz/v-pipe works
Major components
• Input & Configuration:
– Input samples: Notice the “samples/” directory convention (mentioned in the README and documented under “config/README.md”) where the raw sequencing data are organized in a two-level hierarchy with “raw_data” sub-folders.
– Configuration files: The “config” directory contains the main configuration (YAML/HTML) along with virus-specific base configuration files (e.g., hiv, sars-cov-2) that determine which parameters to use.
– Utility scripts: Under “utils/” there are mass-importers and quick installer scripts.
• The Workflow Engine:
– Core Snakemake engine: The “workflow/” directory is the heart of the pipeline. It includes the main Snakefile, environment definitions (conda YAML files under “workflow/envs”), rule definitions (in “workflow/rules”), and schema validations.
– Auxiliary workflows: Also note that in “resources/auxiliary_workflows” there are benchmarking and multi-setup workflows that complement the main pipeline.
• Processing Modules:
– Preprocessing and Quality Assessment: Tools like FastQC, PRINSEQ, and preprocessing rules defined in “workflow/rules/preprocessing.smk”.
– Read Alignment and Reference Handling: Rules for alignment (using ngshmmalign, bwa, minimap, etc.) and consensus building; reference data provided in “resources/” (with virus-specific fasta files, gff/gff3, and primers).
– Variant Calling & Haplotype Reconstruction: SNV calling (likely using ShoRAH, LoFreq, etc.), local and global haplotype assembly (e.g., using Haploclique, SAVAGE) – rules are organized in individual Snakemake files.
– Postprocessing & Visualization: Rules and scripts for generating metrics, statistics, and visualizations in “workflow/scripts” and the “docs/” directory.
• Deployment/Execution & CI Infrastructure:
– Installation/Docker: The README details different deployment methods (quick install, Docker container, Snakedeploy). The Dockerfile at the root and GitHub workflows in “.github/workflows” are also present.
– Testing and Validation: The “tests/” folder contains sample data, test scripts, and expected outputs for regression tests.
– Continuous Integration: There are GitHub workflows (e.g., mega-linter.yml, test-docker.yaml, run_regression_tests.yaml) that enforce code quality, linting, and regression testing.
Relationships and Interactions
– User Input → Configuration:
• The user organizes input files (via the “samples/” directory) and customizes the working configuration (config.yaml) which sets virus-specific parameters.
– Configuration → Workflow Engine:
• Snakemake loads the configuration and selects task rules accordingly.
– Workflow Engine → Processing Modules:
• The central Snakefile orchestrates the execution of preprocessing, alignment, variant calling, haplotype reconstruction and visualization. Each module calls specific external computational tools (like FastQC, ngshmmalign, and others) using dedicated conda environments.
– Auxiliary and CI Processes:
• Auxiliary workflows (benchmarking, multi_setup) are used for performance evaluation and further method definitions stored under “resources/auxiliary_workflows”.
• The GitHub workflows automate testing, container deploys, and linting.
– Output Production:
• Final outputs (consensus sequences, variant call files like VCFs, visualization files, quality assurance metrics) are produced and saved in designated output directories as defined in the configuration.
Architectural Patterns & Design Principles
– Modular & Extensible: The pipeline splits into many small and independent rules (Snakemake rules) that can be easily maintained or extended.
– Environment Isolation: Each rule is executed in its own conda environment (or Docker container) ensuring reproducibility.
– Separation of Concern & Config-Driven: The use of dedicated configuration files (per virus and per environment) separates execution logic from the parameters and sample-specific details.
– CI/CD Integration: GitHub Actions ensure that testing and deployment are automated.
Technologies & Dependencies
– Snakemake as the workflow manager is central to the orchestration.
– Conda (and mamba) environments are used for dependency management.
– Docker is implemented as an alternative deployment mode.
– Various external tools (FastQC, ngshmmalign, bwa, MAFFT, etc.) are managed by Snakemake rules.
– Standard languages: Python and shell scripts are used in “workflow/scripts” and utils.
Details Specific to V‐pipe
– Emphasize virus-specific configuration (with base configs for HIV, SARS-CoV-2, etc.) and note where reference files and primers are stored (in the “resources/” subdirectory).
– Represent that the pipeline can be deployed in multiple ways (locally, via Docker, or using Snakedeploy) as described in the README.
– Detail that there is robust testing and CI infrastructure (with GitHub workflows, regression tests from “tests/” folder).
– The User taking input (sample directory and configuration files)
– The central Snakemake engine (with its main Snakefile, rules and conda environments)
– Sequentially arranged processing modules (preprocessing, alignment, variant calling, haplotype reconstruction, visualization)
– Auxiliary workflows for benchmarking and testing
– Deployment paths (quick install script, Docker, Snakedeploy)
– External dependencies linked to each processing module