How martenj/digestiflow-demux works
Project Type and Purpose
• Type: A Python-based CLI tool that automates demultiplexing of Illumina flowcells.
• Purpose:
– Query a remote Digestiflow Web REST API for run metadata and sample sheets.
– Generate and execute a Snakemake workflow that calls external bioinformatics tools (bcl2fastq, Picard, FastQC, MultiQC) inside isolated Conda environments.
– Post back status, logs, and reports to the Digestiflow Web API.
File Structure
• Top-level:
– digestiflow_demux/: main Python package
– snakemake_support.py, workflow.py, api_client.py: core modules
– wrappers/: per-tool wrapper modules, each with its own environment.yaml and wrapper.py
– tests/: unit-test suite
– CI/config: .github/workflows/build.yml, Makefile, setup.py, requirements/
• Patterns:
– Modular “wrapper” pattern: each external tool is wrapped in a Python interface + dedicated Conda environment.
– Snakemake integration: dynamic generation of a Snakefile and invocation via snakemake_support.py.
Extract Key Architecture Notes from the README
• Configuration layer: reads ~/.digestiflowrc.toml
• CLI entrypoint (__main__.py) calls into workflow.py
• API interactions: api_client.py handles REST calls to get flowcell info, sample sheets, and to post back results and reports
• Workflow orchestration: sample sheet creation → working directory setup → call Snakemake → run tools → collect logs/reports → cleanup → API update
• Optional cluster integration: DRMAA or custom Snakemake jobscript via configuration
Main Components and Their Interactions (to show
a. CLI Frontend
– digestiflow-demux command parses arguments & loads user config
b. Configuration Loader
– Reads [web] and [demux] settings from ~/.digestiflowrc.toml
c. API Client
– GET flowcell metadata & sample sheets
– POST status updates, attachments (reports, logs)
d. Workflow Builder
– workflow.py & snakemake_support.py generate a Snakefile and local working directory
e. Snakemake Engine
– Invoked as a subprocess; orchestrates jobs
f. Wrapper Modules (per tool)
– bcl2fastq, bcl2fastq2, Picard, FastQC, MultiQC
– Each wrapper defines:
• a Conda environment (environment.yaml)
• Python glue code (wrapper.py) that Snakemake calls
g. External Bioinformatics Tools
– Installed inside their Conda envs, invoked by Snakemake
h. Local & Cluster Execution
– Local multithreading or DRMAA/cluster submission
i. Results & Cleanup
– MD5 sums, QC reports, log files
– Working directory teardown (optionally kept)
Architectural Patterns & Design Principles
• Modular wrappers for each external tool, encapsulating dependencies
• Separation of concerns:
– CLI & config parsing
– API communication
– Workflow definition (Snakefile)
– Execution engine (Snakemake)
• Cyber-repeatable environments via Conda YAMLs
• Optional cluster extensibility via DRMAA
Final Checklist
• Clear, descriptive labels on all components
– CLI & config
– API client
– Workflow builder & Snakemake
– Wrappers vs. tools
– External services