# clinical-genomics-laboratory/nf-cgl-cgs architecture

> Architecture diagram of the GitHub repository clinical-genomics-laboratory/nf-cgl-cgs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/clinical-genomics-laboratory/nf-cgl-cgs
- Repository: https://github.com/clinical-genomics-laboratory/nf-cgl-cgs

## Overview

##### Project Type and Purpose
- This is an nf-core/Nextflow DSL2 bioinformatics pipeline named “nf-cgl-cgs” designed to perform Illumina DRAGEN-based alignment and joint genotyping on whole-genome samples. It can either demultiplex raw Illumina runs or pick up from a pre-computed `fastq_list.csv`.

##### High-Level Architecture
- Orchestrator: Nextflow engine invoked via `main.nf` + `nextflow.config`.
- Modules: Reusable process definitions under `modules/local` and `modules/nf-core`.
- Subworkflows: Composite workflows in `subworkflows/local` and `subworkflows/nf-core` that wrap multiple modules into logical steps.
- Helper scripts & configs: Python helpers in `bin/`, static assets/templates in `assets/`, and configuration files in `conf/`.
- Execution environments: Docker/Singularity containers specified in module environment files or via profiles (`-profile docker/singularity/institute`).
- CI/CD & infrastructure as code: GitHub Actions workflows, `.devcontainer`, `.gitpod.yml`, `.nf-core.yml`, and Tower/YML for Pipeline-as-Code.

##### Main Components
a. Nextflow Engine (main.nf + config)
b. Input Sources
- Illumina run directory → demux module (`modules/local/dragen_demultiplex.nf`)
- Or `fastq_list.csv` for already demultiplexed data
c. Core Processing Modules (DSL2 processes)
- Alignment: `modules/local/dragen_align.nf`
- Joint small-variant genotyping: `modules/local/dragen_joint_small_variants.nf`
- (Optional) Joint CNV/SV: `modules/local/dragen_joint_cnv.nf` & `modules/local/dragen_joint_sv.nf`
- Split per-sample VCF: `modules/local/bcftools_split_vcf.nf`
d. Auxiliary Modules
- QC parsing (`parse_qc_metrics.nf`)
- Data transfer (`transfer_data_aws.nf`)
- Samplesheet creation/conversion
e. Subworkflows
f. External Services & Outputs
- AWS S3 (data transfer)
- MultiQC HTML reports
- Final VCF outputs per sample
- Slack/email notifications via templates in `assets/`

##### Relationships & Data Flows
- From demux or fastq list → “Alignment Module” → “Joint Genotyping Subworkflow” → “VCF Splitting Module”
- QC metrics feed into reporting
- Use Docker or Singularity at each module; show container engine as a border around core modules.
- CI/CD pipelines trigger linting, testing, and release workflows but are orthogonal to data flow.

##### Architectural Patterns & Principles
- DSL2 modular design: each process isolated, reusable, and containerized
- Subworkflow composition: higher-level workflows group modules logically
- Infrastructure-as-code: configuration files (`conf/`) and CI pipelines define environment and quality gates
- Separation of concerns: processing vs. reporting vs. data transfer vs. notification

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Nextflow Core
    subgraph "Nextflow Core" 
        direction TB
        NF["Nextflow Engine\n(main.nf + nextflow.config)"]:::core
    end

    %% Input Sources
    subgraph "Input Sources" 
        direction TB
        Illumina["Illumina Run Directory"]:::input
        FastqList["FASTQ List (fastq_list.csv)"]:::input
    end

    %% Demultiplexing Subworkflow
    subgraph "Demultiplexing Subworkflow" 
        direction TB
        DemuxSW["Demultiplexing Subworkflow"]:::module
        DemuxModule["Dragen Demultiplex Module"]:::module
        PrepareDemux["Prepare Demux Script"]:::module
    end

    %% Core Processing Modules
    subgraph "Core Processing" 
        direction TB
        Align["Alignment Module"]:::module
        JointSV["Joint SV Genotyping\n(Optional)"]:::module
        JointCNV["Joint CNV Genotyping\n(Optional)"]:::module
        JointSmall["Joint Small-Variant Genotyping"]:::module
        SplitVCF["Split per-sample VCF"]:::module
    end

    %% Auxiliary Modules
    subgraph "Auxiliary Modules & Helpers" 
        direction TB
        ParseQC["QC Parsing Module"]:::module
        ParseQCScript["QC Parsing Script"]:::module
        TransferAWS["AWS Data Transfer Module"]:::module
        CreateSheet["Create Samplesheet Module"]:::module
        ConvertXLSX["Convert XLSX to CSV Module"]:::module
        CheckSheet["Check Samplesheet Script"]:::module
    end

    %% Reporting & Notifications
    subgraph "Reporting & Notifications" 
        direction TB
        MultiQC["MultiQC Reporting"]:::module
        EmailTemp["Email Templates"]:::module
        SlackRep["Slack Reporting"]:::module
    end

    %% External Services & Outputs
    subgraph "External Services & Outputs" 
        direction TB
        DRAGEN["DRAGEN Bio-IT Platform"]:::external
        S3["AWS S3"]:::external
        Reports["MultiQC HTML Reports"]:::output
        FinalVCF["Final VCF Outputs"]:::output
    end

    %% CI/CD and Infra
    subgraph "CI/CD & Infrastructure" 
        direction TB
        CI["CI Pipelines"]:::infra
        Infra["Infrastructure Configs"]:::infra
    end

    %% Data Flow
    NF -->|"invokes"| DemuxSW
    Illumina -->|"--illumina_rundir"| DemuxSW
    FastqList -->|"--fastq_list"| DemuxSW
    DemuxSW -->|FASTQ| Align
    Align -->|BAM| JointSmall
    JointSmall -->|"gVCF"| SplitVCF
    JointSmall -->|gVCF| JointCNV
    JointSmall -->|gVCF| JointSV
    SplitVCF -->|VCF| FinalVCF
    JointCNV -->|CNV Calls| FinalVCF
    JointSV -->|SV Calls| FinalVCF
    JointSmall -->|QC Metrics| ParseQC
    ParseQC -->|metrics| MultiQC
    TransferAWS -->|upload| S3
    MultiQC -->|report| Reports
    Reports -->|delivery| EmailTemp
    Reports -->|delivery| SlackRep
    NF -->|"profiles"| Infra
    CI -->|"triggers"| NF

    Align -->|uses| DRAGEN
    JointSmall -->|uses| DRAGEN
    JointCNV -->|uses| DRAGEN
    JointSV -->|uses| DRAGEN

    %% Click Events
    click NF "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/main.nf"
    click NF "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/nextflow.config"
    click DemuxModule "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/dragen_demultiplex.nf"
    click DemuxSW "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/subworkflows/local/demultiplex.nf"
    click PrepareDemux "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/bin/prepare_dragen_demux.py"
    click FastqList "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/assets/stub/demux_fastq/Reports/fastq_list.csv"
    click Align "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/dragen_align.nf"
    click JointSmall "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/dragen_joint_small_variants.nf"
    click JointCNV "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/dragen_joint_cnv.nf"
    click JointSV "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/dragen_joint_sv.nf"
    click SplitVCF "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/bcftools_split_vcf.nf"
    click ParseQC "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/parse_qc_metrics.nf"
    click ParseQCScript "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/bin/parse_qc_metrics.py"
    click TransferAWS "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/transfer_data_aws.nf"
    click CreateSheet "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/create_demultiplex_samplesheet.nf"
    click ConvertXLSX "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/local/convert_xlsx_to_csv.nf"
    click CheckSheet "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/bin/check_samplesheet.py"
    click MultiQC "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/modules/nf-core/multiqc/main.nf"
    click EmailTemp "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/assets/email_template.html"
    click SlackRep "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/assets/slackreport.json"
    click CI "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.github/workflows/ci.yml"
    click CI "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.github/workflows/linting.yml"
    click CI "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.github/workflows/awsfulltest.yml"
    click Infra "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.nf-core.yml"
    click Infra "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/tower.yml"
    click Infra "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.devcontainer/devcontainer.json"
    click Infra "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/.gitpod.yml"
    click Infra "https://github.com/clinical-genomics-laboratory/nf-cgl-cgs/blob/main/conf/*.config"

    %% Styles
    classDef core fill:#BBDEFB,stroke:#0D47A1,stroke-width:2px
    classDef module fill:#C8E6C9,stroke:#1B5E20,stroke-width:1px
    classDef external fill:#FFE0B2,stroke:#E65100,stroke-width:2px
    classDef input fill:#E3F2FD,stroke:#1976D2,stroke-width:1px
    classDef output fill:#FFF3E0,stroke:#FB8C00,stroke-width:1px
    classDef infra fill:#ECEFF1,stroke:#37474F,stroke-width:1px
```
