How sanger-tol/variantcalling works
Project Type and Purpose
• This project is a bioinformatics pipeline for variant calling. It processes input PacBio aligned or unaligned reads (CRAM/BAM files) along with a reference FASTA file. The pipeline splits the FASTA into individual sequences, optionally aligns reads, filters for primary alignments, executes variant calling using DeepVariant, then merges the resulting VCF/GVCF files and computes statistics.
• The README emphasizes best-practices, containerization (Docker/Singularity/etc), and modularity (many nf-core modules integrated) using Nextflow DSL2.
Analyzing the File Structure
• Top-Level Files/Directories:
• The presence of directories like “conf”, “docs”, “lib”, “modules”, “subworkflows”, and “workflows” signals a modular Nextflow pipeline.
• "main.nf" at the root is the primary workflow script.
• “modules” contains a host of nf-core modules (e.g., bcftools, blast, cat, deepvariant) organized in subdirectories with environment files, configuration files, and tests. This indicates that the pipeline is built to link together many modular steps.
• “subworkflows” house additional, possibly optional, workflow snippets (like aligning and filtering) that are integrated into the primary workflow.
• “lib” includes Groovy scripts (e.g., WorkflowMain, WorkflowVariantcalling, Utils) — these scripts are likely custom code supporting the pipeline’s logic.
• Standard files such as nextflow.config, Docker/Conda profiles, and CI workflows in “.github” reflect production-readiness and community integration with nf-core.
• Architectural Pattern:
• The pipeline leverages the Nextflow DSL2 modular design, which organizes processes into distinct modules and subworkflows.
• Containerization is used to encapsulate each process individually for reproducibility.
• An MVC-like split is evident where Nextflow manages the workflow orchestration (controller), the process scripts and modules act as the business logic (model), and output reporting/documentation comprises the view.
a. Primary Components:
• User/Trigger: The entry point where a user submits parameters (input samplesheet, FASTA, etc.) through a Nextflow run command.
• Nextflow Workflow Engine: Core orchestrator represented by “main.nf” and the configuration/system files (nextflow.config, profiles in “conf”).
• Input Data Nodes: The samplesheet CSV and FASTA genome file.
• Preprocessing Steps:
• Fasta splitter (splits the FASTA file into individual sequence files).
• Read alignment (if needed) and merge steps (merging with same sample names), indicated in the README.
• Filtering Process: The step that applies the “-F 0x900” filter to remove secondary alignments.
• Variant Calling (DeepVariant processes): Modules that run variant calling using DeepVariant via containerized processes.
• Post-processing: Modules for merging VCF/GVCF outputs (and subsequent processing using VCFtools for heterozygosity and diversity analysis).
• Software Dependency & Container Management: Docker, Singularity, Conda, etc., which manage the environment for each process. Represent this as an underpinning “container layer.”
• External Integrations: nf-core/modules that are reused and community-curated; also Nextflow Tower for launching jobs.
b. Relationships and Data Flow:
• Dependency between configuration profiles (from “conf” and “nextflow.config”) and container invocation (Docker/Singularity profile selection).
Architectural Patterns and Design Principles
• Containerization: Illustrate the container layer (using Docker/Singularity symbols) associated with each processing module.
• Data Flow Orientation: Emphasize the sequential pipeline steps (fasta splitting → alignment/filtering → DeepVariant execution → VCF merge → VCFtools analysis).
• Testing & CI: Optionally include a component that represents the automated testing and CI workflows (from the “.github/workflows” folder).
Emphasizing Essential Architectural Elements
• Starting with user input and configuration profiles.
• Nextflow workflow orchestration that invokes modules and subworkflows.
• The containerized execution of processes (ensuring reproducibility).
• The integration of modular nf-core components along with custom scripts in “lib”.
• How each processing step connects to its subsequent component so that the full variant calling process (from reading inputs to generating final statistical reports) is visible.