How nf-core/sarek works
Project Type & Purpose
• nf‐core/sarek is a bioinformatics workflow pipeline built using Nextflow. Its purpose is to process raw sequencing data (fastq) to produce variant calls. It encompasses quality control, trimming, mapping, BAM processing, variant calling, filtering/annotation, and QC summarization.
• The README clearly documents the pipeline’s capabilities, its use of containerization (Docker/Singularity), continuous integration, and integration with nf‐core modules. This suggests a modular design where each analysis step is an independent process.
File Structure
• The directory structure indicates that the pipeline is built as a collection of modular processes (each one as its own Nextflow process) which are connected by shared configuration and data channels.
• There are also directories for tests, CI (under .github/workflows and others), and additional resources (assets, posters, etc.) – emphasizing the pipeline’s rigorous testing and documentation.
Extract Insights from the README
• The README highlights multiple essential pipeline steps: fastq QC/trimming, mapping (BWA/Dragmap/Sentieon), BAM processing, multiple variant callers (ASCAT, CNVkit, DeepVariant, Mutect2, etc.), variant filtering/annotation (SnpEff, VEP, BCFtools annotations), and QC summarization (MultiQC).
• It also emphasizes containerization, reproducibility, and execution across different compute infrastructures.
a. Main Components:
– Workflow Engine: Nextflow (acts as the orchestrator)
– Input: Samplesheet (CSV with fastq file metadata)
– Data Preparation & QC: FastQC, fastp, trimming modules (e.g., using “assets” for templates or configurations)
– Read Mapping: Processes using BWA-mem, BWA-mem2, dragmap, or Sentieon aligner; represented as distinct modules
– BAM Processing: Modules for marking duplicates, base recalibration (GATK or Sentieon alternatives)
– Variant Calling: Multiple variant caller modules (ASCAT, CNVkit, Control-FREEC, DeepVariant, freebayes, Mutect2, etc.)
– Variant Filtering & Annotation: SnpEff, Ensembl VEP, BCFtools annotation modules
– QC Summarization: MultiQC and other summarizing/reporting modules
– Output: A set of result files (VCF files, reports, logs) stored in an output directory
b. Relationships & Data Flow:
– Connect the QC outputs to the mapping stage.
– From mapping, show parallel or sequential flows to BAM processing.
– Subsequent annotation, filtering, and summarization follow the calling step.
– Once complete, outputs (VCF, QC reports, dashboards) are collected.
c. Architectural Patterns & Principles:
– Modularity: Each pipeline stage is encapsulated in its own module (see “modules/nf-core” and “modules/local” directories).
– Containerization: Each process runs within its own container (Docker/Singularity) for reproducibility.
– Data-Driven Flow: Processes communicate via channels (data streams) in Nextflow DSL2, so represent explicit data flows between components.
– High Reproducibility & Scalability: Continuous integration (referenced in .github/workflows) and automated testing underpin the workflow’s robustness.
– Extensibility: Use of shared nf-core modules and subworkflows demonstrates the design’s emphasis on code reuse and easy updates.
d. Relevant Technologies:
– Nextflow DSL2: central orchestration technology.
– Docker, Singularity, Conda: handling container management and software dependency versions.
– Bioinformatics tools: FastQC, fastp, BWA variants, GATK, Sentieon, MultiQC, etc.
– Configuration & Testing: nf-core config files, CI/CD workflows, and extensive test suites.