How clinical-genomics-laboratory/nf-cgl-cgs works
Project Type and Purpose
• This is an nf-core/Nextflow DSL2 bioinformatics pipeline named “nf-cgl-cgs” designed to perform Illumina DRAGEN-based alignment and joint genotyping on whole-genome samples. It can either demultiplex raw Illumina runs or pick up from a pre-computed fastq_list.csv.
High-Level Architecture
• Orchestrator: Nextflow engine invoked via main.nf + nextflow.config.
• Modules: Reusable process definitions under modules/local and modules/nf-core.
• Subworkflows: Composite workflows in subworkflows/local and subworkflows/nf-core that wrap multiple modules into logical steps.
• Helper scripts & configs: Python helpers in bin/, static assets/templates in assets/, and configuration files in conf/.
• Execution environments: Docker/Singularity containers specified in module environment files or via profiles (-profile docker/singularity/institute).
• CI/CD & infrastructure as code: GitHub Actions workflows, .devcontainer, .gitpod.yml, .nf-core.yml, and Tower/YML for Pipeline-as-Code.
Main Components
a. Nextflow Engine (main.nf + config)
b. Input Sources
• Illumina run directory → demux module (modules/local/dragen_demultiplex.nf)
• Or fastq_list.csv for already demultiplexed data
c. Core Processing Modules (DSL2 processes)
• Alignment: modules/local/dragen_align.nf
• Joint small-variant genotyping: modules/local/dragen_joint_small_variants.nf
• (Optional) Joint CNV/SV: modules/local/dragen_joint_cnv.nf & modules/local/dragen_joint_sv.nf
• Split per-sample VCF: modules/local/bcftools_split_vcf.nf
d. Auxiliary Modules
• QC parsing (parse_qc_metrics.nf)
• Data transfer (transfer_data_aws.nf)
• Samplesheet creation/conversion
e. Subworkflows
f. External Services & Outputs
• AWS S3 (data transfer)
• MultiQC HTML reports
• Final VCF outputs per sample
• Slack/email notifications via templates in assets/
Relationships & Data Flows
• From demux or fastq list → “Alignment Module” → “Joint Genotyping Subworkflow” → “VCF Splitting Module”
• QC metrics feed into reporting
• Use Docker or Singularity at each module; show container engine as a border around core modules.
• CI/CD pipelines trigger linting, testing, and release workflows but are orthogonal to data flow.
Architectural Patterns & Principles
• DSL2 modular design: each process isolated, reusable, and containerized
• Subworkflow composition: higher-level workflows group modules logically
• Infrastructure-as-code: configuration files (conf/) and CI pipelines define environment and quality gates
• Separation of concerns: processing vs. reporting vs. data transfer vs. notification