# nf-core/sarek architecture

> Architecture diagram of the GitHub repository nf-core/sarek, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nf-core/sarek
- Repository: https://github.com/nf-core/sarek

## Overview

##### Project Type & Purpose
- nf‐core/sarek is a bioinformatics workflow pipeline built using Nextflow. Its purpose is to process raw sequencing data (fastq) to produce variant calls. It encompasses quality control, trimming, mapping, BAM processing, variant calling, filtering/annotation, and QC summarization.
- The README clearly documents the pipeline’s capabilities, its use of containerization (Docker/Singularity), continuous integration, and integration with nf‐core modules. This suggests a modular design where each analysis step is an independent process.

##### File Structure
- The directory structure indicates that the pipeline is built as a collection of modular processes (each one as its own Nextflow process) which are connected by shared configuration and data channels.
- There are also directories for tests, CI (under .github/workflows and others), and additional resources (assets, posters, etc.) – emphasizing the pipeline’s rigorous testing and documentation.

##### Extract Insights from the README
- The README highlights multiple essential pipeline steps: fastq QC/trimming, mapping (BWA/Dragmap/Sentieon), BAM processing, multiple variant callers (ASCAT, CNVkit, DeepVariant, Mutect2, etc.), variant filtering/annotation (SnpEff, VEP, BCFtools annotations), and QC summarization (MultiQC).
- It also emphasizes containerization, reproducibility, and execution across different compute infrastructures.

a. Main Components:
– Workflow Engine: Nextflow (acts as the orchestrator)
– Input: Samplesheet (CSV with fastq file metadata)
– Data Preparation & QC: FastQC, fastp, trimming modules (e.g., using “assets” for templates or configurations)
– Read Mapping: Processes using BWA-mem, BWA-mem2, dragmap, or Sentieon aligner; represented as distinct modules
– BAM Processing: Modules for marking duplicates, base recalibration (GATK or Sentieon alternatives)
– Variant Calling: Multiple variant caller modules (ASCAT, CNVkit, Control-FREEC, DeepVariant, freebayes, Mutect2, etc.)
– Variant Filtering & Annotation: SnpEff, Ensembl VEP, BCFtools annotation modules
– QC Summarization: MultiQC and other summarizing/reporting modules
– Output: A set of result files (VCF files, reports, logs) stored in an output directory

b. Relationships & Data Flow:
– Connect the QC outputs to the mapping stage.
– From mapping, show parallel or sequential flows to BAM processing.
– Subsequent annotation, filtering, and summarization follow the calling step.
– Once complete, outputs (VCF, QC reports, dashboards) are collected.

c. Architectural Patterns & Principles:
– Modularity: Each pipeline stage is encapsulated in its own module (see “modules/nf-core” and “modules/local” directories).
– Containerization: Each process runs within its own container (Docker/Singularity) for reproducibility.
– Data-Driven Flow: Processes communicate via channels (data streams) in Nextflow DSL2, so represent explicit data flows between components.
– High Reproducibility & Scalability: Continuous integration (referenced in .github/workflows) and automated testing underpin the workflow’s robustness.
– Extensibility: Use of shared nf-core modules and subworkflows demonstrates the design’s emphasis on code reuse and easy updates.

d. Relevant Technologies:
– Nextflow DSL2: central orchestration technology.
– Docker, Singularity, Conda: handling container management and software dependency versions.
– Bioinformatics tools: FastQC, fastp, BWA variants, GATK, Sentieon, MultiQC, etc.
– Configuration & Testing: nf-core config files, CI/CD workflows, and extensive test suites.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Components
    Engine["Nextflow Orchestrator"]:::orchestrator
    Input["Samplesheet (CSV)"]:::input

    %% Configuration & CI
    subgraph "Configuration & CI"
        GlobalConfig["Global Config"]:::config
        CI["CI/CD & Containerization"]:::config
    end

    %% Data Preparation & QC
    subgraph "Data Preparation & QC"
        FastQC["FASTQ QC (FastQC)"]:::processing
        Fastp["Trimming (fastp)"]:::processing
        TrimConf["Trimming Config"]:::config
    end

    %% Read Mapping
    subgraph "Read Mapping"
        BWA_mem["BWA Mapping (mem)"]:::mapping
        BWA_index["BWA Index"]:::mapping
        Dragmap["Dragmap Mapping"]:::mapping
    end

    %% BAM Processing
    subgraph "BAM Processing"
        MarkDup["MarkDuplicates"]:::processing
        Recalib["Base Recalibration"]:::processing
    end

    %% Variant Calling
    subgraph "Variant Calling"
        ASCAT["ASCAT"]:::variant
        DeepVariant["DeepVariant"]:::variant
        Freebayes["Freebayes"]:::variant
        Mutect2["Mutect2"]:::variant
        CNVkit["CNVkit"]:::variant
        ControlFREEC["Control-FREEC"]:::variant
    end

    %% Variant Filtering & Annotation
    subgraph "Variant Filtering & Annotation"
        SnpEff["SnpEff"]:::annotation
        VEP["Ensembl VEP"]:::annotation
        BCFTools["BCFTools Annotation"]:::annotation
        AnnVariants["Annotated Variants"]:::annotation
    end

    %% QC Summarization & Output
    MultiQC["MultiQC"]:::qc
    Output["Final Outputs (VCF, Reports)"]:::qc

    %% Data Flow
    GlobalConfig --> Engine
    CI --> Engine
    Engine --> Input

    Input --> FastQC
    Input --> Fastp
    Fastp --> TrimConf

    %% Data Preparation to Read Mapping
    FastQC --> BWA_mem
    Fastp --> BWA_mem

    %% Read Mapping internal flow
    BWA_mem --> BWA_index
    BWA_index --> MarkDup
    Dragmap --> MarkDup

    %% BAM Processing flow
    MarkDup --> Recalib

    %% Variant Calling flow
    Recalib --> ASCAT
    Recalib --> DeepVariant
    Recalib --> Freebayes
    Recalib --> Mutect2
    Recalib --> CNVkit
    Recalib --> ControlFREEC

    %% Merge Variant Calling outputs
    ASCAT --> AnnVariants
    DeepVariant --> AnnVariants
    Freebayes --> AnnVariants
    Mutect2 --> AnnVariants
    CNVkit --> AnnVariants
    ControlFREEC --> AnnVariants

    %% Annotation to QC Summarization & Output
    AnnVariants --> SnpEff
    AnnVariants --> VEP
    AnnVariants --> BCFTools
    SnpEff --> MultiQC
    VEP --> MultiQC
    BCFTools --> MultiQC
    MultiQC --> Output

    %% Styles
    classDef orchestrator fill:#ffe0e0,stroke:#b22222,stroke-width:2px;
    classDef input fill:#e0ffe0,stroke:#228b22,stroke-width:2px;
    classDef processing fill:#e0e0ff,stroke:#4169e1,stroke-width:2px;
    classDef mapping fill:#fff0e0,stroke:#ff8c00,stroke-width:2px;
    classDef variant fill:#e6ffe6,stroke:#32cd32,stroke-width:2px;
    classDef annotation fill:#f0e6ff,stroke:#8a2be2,stroke-width:2px;
    classDef qc fill:#ffffe0,stroke:#daa520,stroke-width:2px;
    classDef config fill:#f5f5dc,stroke:#a0522d,stroke-width:2px;

    %% Click Events
    click Engine "https://github.com/nf-core/sarek/blob/master/main.nf"
    click Input "https://github.com/nf-core/sarek/blob/master/assets/samplesheet.csv"
    click FastQC "https://github.com/nf-core/sarek/tree/master/modules/nf-core/fastqc"
    click Fastp "https://github.com/nf-core/sarek/tree/master/modules/nf-core/fastp"
    click TrimConf "https://github.com/nf-core/sarek/blob/master/conf/modules/trimming.config"
    click BWA_mem "https://github.com/nf-core/sarek/tree/master/modules/nf-core/bwa/mem"
    click BWA_index "https://github.com/nf-core/sarek/tree/master/modules/nf-core/bwa/index"
    click Dragmap "https://github.com/nf-core/sarek/tree/master/modules/nf-core/dragmap"
    click MarkDup "https://github.com/nf-core/sarek/blob/master/conf/modules/markduplicates.config"
    click MarkDup "https://github.com/nf-core/sarek/tree/master/subworkflows/local/bam_markduplicates"
    click Recalib "https://github.com/nf-core/sarek/blob/master/conf/modules/prepare_recalibration.config"
    click Recalib "https://github.com/nf-core/sarek/tree/master/subworkflows/local/bam_baserecalibrator"
    click ASCAT "https://github.com/nf-core/sarek/tree/master/modules/nf-core/ascat"
    click DeepVariant "https://github.com/nf-core/sarek/tree/master/modules/nf-core/deepvariant/rundeepvariant"
    click Freebayes "https://github.com/nf-core/sarek/tree/master/modules/nf-core/freebayes"
    click Mutect2 "https://github.com/nf-core/sarek/tree/master/modules/nf-core/gatk4/mutect2"
    click CNVkit "https://github.com/nf-core/sarek/tree/master/modules/nf-core/cnvkit"
    click ControlFREEC "https://github.com/nf-core/sarek/tree/master/modules/nf-core/controlfreec"
    click SnpEff "https://github.com/nf-core/sarek/tree/master/modules/nf-core/snpeff/snpeff"
    click VEP "https://github.com/nf-core/sarek/tree/master/modules/nf-core/ensemblvep/vep"
    click BCFTools "https://github.com/nf-core/sarek/tree/master/modules/nf-core/bcftools/annotate"
    click MultiQC "https://github.com/nf-core/sarek/tree/master/modules/nf-core/multiqc"
    click GlobalConfig "https://github.com/nf-core/sarek/tree/master/conf"
    click CI "https://github.com/nf-core/sarek/tree/master/.github/workflows"
    click CI "https://github.com/nf-core/sarek/blob/master/.devcontainer"
```
