# xwanglabthu/cfdnapipe architecture

> Architecture diagram of the GitHub repository xwanglabthu/cfdnapipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xwanglabthu/cfdnapipe
- Repository: https://github.com/xwanglabthu/cfdnapipe

## Overview

##### Project type and purpose
- This is a Python‐based, command‐line cfDNA analysis toolkit (“cfDNApipe”) implementing an end‐to‐end WGS/WGBS data processing pipeline.
- It wraps dozens of external bioinformatics tools (FastQC, Bowtie2, Bismark, GATK, Centrifuge, samtools, etc.) into Python “Fun_*” modules and orchestrates them via a DAG (directed acyclic graph) of pipeline steps.
- Supports single‐group and case–control workflows, automated reference download/build, breakpoint detection, parallel execution, and report generation.

##### File structure
- Top-level scripts & config: setup.py, environment.yml, Pipeline.py, Configure.py/Configure2.py.
- cfDNApipe/ directory:
- StepBase.py & cfDNA_utils.py (core pipeline‐engine and utilities)
- Configure.py/Configure2.py (global config holders for single vs case–control)
- Fun_*.py (one file per pipeline step, e.g., Fun_bowtie2.py, Fun_mutect2t.py, Fun_deconvolution.py, etc.)
- report_generator*.py (HTML report builder)
- demo/ & test/: example scripts, input/output snapshots, functional and pipeline tests, and demo report results.
- docs/ & pics/: detailed user docs and flowcharts.
- External dependencies managed by Conda (environment.yml).

##### README insights
- Describes high‐level flowchart for WGS/WGBS with numbered steps (“step_01_inputprocess” → “step_02_fastqc” → … → final_result + report_result).
- Explains two global “pipeConfigure” entry points (single vs case–control), and two Python invocations (“cfDNAWGBS()” or “cfDNAWGBS2()”).
- Emphasizes the dataflow‐graph principle: every Fun_* returns an object representing outputs, which feed into the next Fun_* via the “upstream” argument or explicit parameters.

##### Relationships & interactions
Input FASTQ → Step_01_inputprocess → Step_02_fastqc → … → Step_n → final_result & report_result
- Global config object feeds into every step (threads, genome, refdir, outdir).
- Optional branches:
- Case vs Control sub‐pipelines (two parallel DAGs under “case/” and “control/” subfolders).

##### Architectural patterns & principles
- Dataflow/DAG orchestration: each module defines inputs/outputs and dependencies.
- “Configure” singleton holds global parameters—Dependency Injection pattern.
- Breakpoint detection via MD5 hashing of inputs/outputs—Idempotent, resumable execution.
- Plugin‐style modules: each Fun_*.py registers itself as a pipeline node.
- Parallelism: per‐sample and per‐chromosome splitting, coordinating via threads/process pools.

##### Detail
- Folder structure outcomes: intermediate_result/step_##_*, final_result, report_result.
- How Configure vs Configure2 selects single vs multi‐sample vs case/control folder layouts.
- Auxiliary mechanisms: breakpoint detection, automatic reference download, parallel splitting.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "cfDNApipe Python API"
        entry1["pipeConfigure()/cfDNAWGBS()"]:::blue
        entry2["pipeConfigure2()/cfDNAWGBS2()"]:::blue
    end
    entry1 --> PM
    entry2 --> PM

    PM["Pipeline Manager\n(Pipeline.py)"]:::blue
    click PM "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Pipeline.py"

    subgraph "Configuration"
        Conf1["Configure (single)"]:::blue
        Conf2["Configure2 (case–control)"]:::blue
    end
    click Conf1 "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Configure.py"
    click Conf2 "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Configure2.py"

    PM --> Conf1
    PM --> Conf2

    subgraph "Core Engine"
        StepBase["StepBase"]:::blue
        Utils["cfDNA_utils"]:::blue
    end
    click StepBase "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/StepBase.py"
    click Utils "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/cfDNA_utils.py"

    PM --> StepBase
    PM --> Utils

    subgraph "Reference Build"
        RefScript["src_href.bz2"]:::blue
    end
    click RefScript "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/data/src_href.bz2"
    RefScript --> REF

    subgraph "Data Stores"
        FASTQ["Input FASTQ\n(folder)"]:::green
        REF["refdir\n(genome, bisulfite, variant)"]:::green
        INT["intermediate_result"]:::green
        FINAL["final_result"]:::green
        REPORT_DS["report_result"]:::green
    end

    subgraph "Pipeline Steps"
        Fun_input["Fun_inputProcess"]:::blue
        Fun_fastqc["Fun_fastqc"]:::blue
        Fun_adapter["Fun_adapterremoval"]:::blue
        Fun_bowtie2["Fun_bowtie2\n(Align)"]:::blue
        Fun_bismark["Fun_bismark\n(Bisulphite Align)"]:::blue
        Fun_bismarkdedup["Fun_bismarkdedup"]:::blue
        Fun_bismarkmeth["Fun_bismarkmethylex"]:::blue
        Fun_baseRecal["Fun_baseRecalibrator & BQSR"]:::blue
        Fun_mutect2t["Fun_mutect2t"]:::blue
        Fun_mutect2n["Fun_mutect2n"]:::blue
        Fun_gatherVCF["Fun_gatherVCF & filterMutectCalls"]:::blue
        Fun_CNV["Fun_CNV modules"]:::blue
        Fun_fragLen["Fun_fragLen modules"]:::blue
        Fun_OCF["Fun_OCF modules"]:::blue
        Fun_deconv["Fun_deconv"]:::blue
        Fun_other["Analytics (DMR, PCA, GCcorrect, contamination, virus)"]:::blue
    end

    click Fun_input "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_inputProcess.py"
    click Fun_fastqc "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_fastqc.py"
    click Fun_adapter "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_adapterremoval.py"
    click Fun_bowtie2 "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_bowtie2.py"
    click Fun_bismark "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_bismark.py"
    click Fun_bismarkdedup "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_bismarkdedup.py"
    click Fun_bismarkmeth "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_bismarkmethylex.py"
    click Fun_baseRecal "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_baseRecalibrator.py"
    click Fun_mutect2t "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_mutect2t.py"
    click Fun_mutect2n "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_mutect2n.py"
    click Fun_gatherVCF "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_gatherVCF.py"
    click Fun_CNV "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_CNV.py"
    click Fun_fragLen "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_fragLen.py"
    click Fun_OCF "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_OCF.py"
    click Fun_deconv "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/Fun_deconv.py"

    FASTQ --> Fun_input --> Fun_fastqc --> Fun_adapter --> Fun_bowtie2 --> Fun_bismark --> Fun_bismarkdedup --> Fun_bismarkmeth --> Fun_baseRecal --> Fun_mutect2t --> Fun_mutect2n --> Fun_gatherVCF --> Fun_CNV --> Fun_fragLen --> Fun_OCF --> Fun_deconv --> Fun_other --> FINAL --> REPORT_DS

    subgraph "External Tools" 
        Bowtie2["Bowtie2"]:::orange
        Bismark["Bismark"]:::orange
        GATK["GATK4/Mutect2"]:::orange
        Samtools["samtools"]:::orange
        Centrifuge["Centrifuge"]:::orange
        Rscript["R scripts"]:::orange
    end

    Fun_bowtie2 -->|"shell"| Bowtie2
    Fun_bismark -->|"shell"| Bismark
    Fun_bismarkdedup -->|"shell"| Bismark
    Fun_bismarkmeth -->|"shell"| Bismark
    Fun_baseRecal -->|"shell"| GATK
    Fun_mutect2t -->|"shell"| GATK
    Fun_mutect2n -->|"shell"| GATK
    Fun_gatherVCF -->|"shell"| Samtools
    Fun_other -->|"shell"| Rscript
    Fun_other -->|"shell"| Centrifuge

    subgraph "Report Generation"
        R1["report_generator.py"]:::grey
        R2["report_generator_comp.py"]:::grey
    end
    click R1 "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/report_generator.py"
    click R2 "https://github.com/xwanglabthu/cfdnapipe/blob/master/cfDNApipe/report_generator_comp.py"

    FINAL --> R1
    FINAL --> R2
    REPORT_DS --> R1
    REPORT_DS --> R2

    subgraph "Legend"
        L1["Python Modules"]:::blue
        L2["Data Stores"]:::green
        L3["External Binaries"]:::orange
        L4["Reports"]:::grey
    end

    classDef blue fill:#D0E6FF,stroke:#045FB4,color:#000;
    classDef green fill:#D5F5E3,stroke:#229954,color:#000;
    classDef orange fill:#FCF3CF,stroke:#F1C40F,color:#000;
    classDef grey fill:#F2F2F2,stroke:#AAAAAA,color:#000;
```
