How xwanglabthu/cfdnapipe works
Project type and purpose
• This is a Python‐based, command‐line cfDNA analysis toolkit (“cfDNApipe”) implementing an end‐to‐end WGS/WGBS data processing pipeline.
• It wraps dozens of external bioinformatics tools (FastQC, Bowtie2, Bismark, GATK, Centrifuge, samtools, etc.) into Python “Fun_*” modules and orchestrates them via a DAG (directed acyclic graph) of pipeline steps.
• Supports single‐group and case–control workflows, automated reference download/build, breakpoint detection, parallel execution, and report generation.
File structure
• Top-level scripts & config: setup.py, environment.yml, Pipeline.py, Configure.py/Configure2.py.
• cfDNApipe/ directory:
• StepBase.py & cfDNA_utils.py (core pipeline‐engine and utilities)
• Configure.py/Configure2.py (global config holders for single vs case–control)
• Fun_*.py (one file per pipeline step, e.g., Fun_bowtie2.py, Fun_mutect2t.py, Fun_deconvolution.py, etc.)
• report_generator*.py (HTML report builder)
• demo/ & test/: example scripts, input/output snapshots, functional and pipeline tests, and demo report results.
• docs/ & pics/: detailed user docs and flowcharts.
• External dependencies managed by Conda (environment.yml).
README insights
• Describes high‐level flowchart for WGS/WGBS with numbered steps (“step_01_inputprocess” → “step_02_fastqc” → … → final_result + report_result).
• Explains two global “pipeConfigure” entry points (single vs case–control), and two Python invocations (“cfDNAWGBS()” or “cfDNAWGBS2()”).
• Emphasizes the dataflow‐graph principle: every Fun_* returns an object representing outputs, which feed into the next Fun_* via the “upstream” argument or explicit parameters.
Relationships & interactions
Input FASTQ → Step_01_inputprocess → Step_02_fastqc → … → Step_n → final_result & report_result
• Global config object feeds into every step (threads, genome, refdir, outdir).
• Optional branches:
• Case vs Control sub‐pipelines (two parallel DAGs under “case/” and “control/” subfolders).
Architectural patterns & principles
• Dataflow/DAG orchestration: each module defines inputs/outputs and dependencies.
• “Configure” singleton holds global parameters—Dependency Injection pattern.
• Breakpoint detection via MD5 hashing of inputs/outputs—Idempotent, resumable execution.
• Plugin‐style modules: each Fun_*.py registers itself as a pipeline node.
• Parallelism: per‐sample and per‐chromosome splitting, coordinating via threads/process pools.
Detail
• Folder structure outcomes: intermediate_result/step_##_*, final_result, report_result.
• How Configure vs Configure2 selects single vs multi‐sample vs case/control folder layouts.
• Auxiliary mechanisms: breakpoint detection, automatic reference download, parallel splitting.