# martenj/digestiflow-demux architecture

> Architecture diagram of the GitHub repository martenj/digestiflow-demux, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/martenj/digestiflow-demux
- Repository: https://github.com/martenj/digestiflow-demux

## Overview

##### Project Type and Purpose
- Type: A Python-based CLI tool that automates demultiplexing of Illumina flowcells.
- Purpose:
– Query a remote Digestiflow Web REST API for run metadata and sample sheets.
– Generate and execute a Snakemake workflow that calls external bioinformatics tools (bcl2fastq, Picard, FastQC, MultiQC) inside isolated Conda environments.
– Post back status, logs, and reports to the Digestiflow Web API.

##### File Structure
- Top-level:
– `digestiflow_demux/`: main Python package
– `snakemake_support.py`, `workflow.py`, `api_client.py`: core modules
– `wrappers/`: per-tool wrapper modules, each with its own `environment.yaml` and `wrapper.py`
– `tests/`: unit-test suite
– CI/config: `.github/workflows/build.yml`, `Makefile`, `setup.py`, `requirements/`
- Patterns:
– Modular “wrapper” pattern: each external tool is wrapped in a Python interface + dedicated Conda environment.
– Snakemake integration: dynamic generation of a Snakefile and invocation via `snakemake_support.py`.

##### Extract Key Architecture Notes from the README
- Configuration layer: reads `~/.digestiflowrc.toml`
- CLI entrypoint (`__main__.py`) calls into `workflow.py`
- API interactions: `api_client.py` handles REST calls to get flowcell info, sample sheets, and to post back results and reports
- Workflow orchestration: sample sheet creation → working directory setup → call Snakemake → run tools → collect logs/reports → cleanup → API update
- Optional cluster integration: DRMAA or custom Snakemake jobscript via configuration

##### Main Components and Their Interactions (to show
a. CLI Frontend
– `digestiflow-demux` command parses arguments & loads user config
b. Configuration Loader
– Reads `[web]` and `[demux]` settings from `~/.digestiflowrc.toml`
c. API Client
– GET flowcell metadata & sample sheets
– POST status updates, attachments (reports, logs)
d. Workflow Builder
– `workflow.py` & `snakemake_support.py` generate a Snakefile and local working directory
e. Snakemake Engine
– Invoked as a subprocess; orchestrates jobs
f. Wrapper Modules (per tool)
– bcl2fastq, bcl2fastq2, Picard, FastQC, MultiQC
– Each wrapper defines:
- a Conda environment (`environment.yaml`)
- Python glue code (`wrapper.py`) that Snakemake calls
g. External Bioinformatics Tools
– Installed inside their Conda envs, invoked by Snakemake
h. Local & Cluster Execution
– Local multithreading or DRMAA/cluster submission
i. Results & Cleanup
– MD5 sums, QC reports, log files
– Working directory teardown (optionally kept)

##### Architectural Patterns & Design Principles
- Modular wrappers for each external tool, encapsulating dependencies
- Separation of concerns:
– CLI & config parsing
– API communication
– Workflow definition (Snakefile)
– Execution engine (Snakemake)
- Cyber-repeatable environments via Conda YAMLs
- Optional cluster extensibility via DRMAA

##### Final Checklist
- Clear, descriptive labels on all components
– CLI & config
– API client
– Workflow builder & Snakemake
– Wrappers vs. tools
– External services

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User/CLI Layer
    subgraph "User/CLI Layer"
        CLI["CLI Frontend\n(__main__.py)"]:::frontend
        click CLI "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/__main__.py"
    end

    %% Configuration
    Config["Configuration Loader\n(~/.digestiflowrc.toml)"]:::config
    click Config "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/__main__.py"

    %% API Client
    Api["API Client\n(api_client.py)"]:::api
    click Api "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/api_client.py"

    %% Workflow Builder & Snakemake
    subgraph "Workflow Orchestration" 
        WB1["Workflow Builder\n(workflow.py)"]:::workflow
        WB2["Snakemake Support\n(snakemake_support.py)"]:::workflow
        SNKFILE["Snakefile"]:::workflow
        click WB1 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/workflow.py"
        click WB2 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/snakemake_support.py"
        click SNKFILE "https://github.com/martenj/digestiflow-demux/tree/master/digestiflow_demux/Snakefile"
    end

    Snakemake["Snakemake Engine"]:::engine

    %% Wrapper Modules
    subgraph "Wrapper Modules" 
        BCL["bcl2fastq\nwrapper.py"]:::wrapper
        E1["env"]:::wrapper
        click BCL "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/bcl2fastq/wrapper.py"
        click E1 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/bcl2fastq/environment.yaml"

        BCL2["bcl2fastq2\nwrapper.py"]:::wrapper
        E2["env"]:::wrapper
        click BCL2 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/bcl2fastq2/wrapper.py"
        click E2 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/bcl2fastq2/environment.yaml"

        P1["Picard: basecalls_to_fastq"]:::wrapper
        PE1["env"]:::wrapper
        click P1 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/picard/basecalls_to_fastq/wrapper.py"
        click PE1 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/picard/basecalls_to_fastq/environment.yaml"

        P2["Picard: extract_barcodes"]:::wrapper
        PE2["env"]:::wrapper
        click P2 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/picard/extract_barcodes/wrapper.py"
        click PE2 "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/picard/extract_barcodes/environment.yaml"

        FQC["FastQC\nwrapper.py"]:::wrapper
        EFQC["env"]:::wrapper
        click FQC "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/fastqc/wrapper.py"
        click EFQC "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/fastqc/environment.yaml"

        MQC["MultiQC\nwrapper.py"]:::wrapper
        EMQC["env"]:::wrapper
        click MQC "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/multiqc/wrapper.py"
        click EMQC "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/wrappers/multiqc/environment.yaml"
    end

    %% Execution Paths
    Local["Local Execution"]:::external
    Cluster["Cluster Execution\n(DRMAA)"]:::external

    %% Utilities & Exceptions
    Exc["Exception Handling\n(exceptions.py)"]:::utils
    click Exc "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/exceptions.py"
    Mask["Utility Modules\n(bases_mask.py)"]:::utils
    click Mask "https://github.com/martenj/digestiflow-demux/blob/master/digestiflow_demux/bases_mask.py"

    %% Tests & CI
    Tests["Tests Suite"]:::external
    click Tests "https://github.com/martenj/digestiflow-demux/tree/master/tests/"
    CI["CI & Build Config"]:::external
    click CI "https://github.com/martenj/digestiflow-demux/blob/master/.github/workflows/build.yml"

    %% External Service
    Web["Digestiflow Web REST API"]:::external

    %% Connections
    CLI -->|"load config"| Config
    Config -->|"GET settings"| CLI
    CLI -->|"invoke"| Api
    Api -->|"GET metadata & sample sheet"| WB1
    WB1 -->|"generate Snakefile"| WB2
    WB2 -->|"write"| SNKFILE
    WB2 -->|"invoke"| Snakemake
    SNKFILE --> Snakemake
    Snakemake -->|"run wrappers"| BCL & BCL2 & P1 & P2 & FQC & MQC
    Snakemake --> Local
    Snakemake --> Cluster
    BCL & BCL2 & P1 & P2 & FQC & MQC -->|"tools exec"| Web
    Snakemake -->|"POST status+reports"| Api
    Api -->|"POST results"| Web

    %% Class Definitions
    classDef frontend fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef config fill:#E0F8E0,stroke:#333,stroke-width:1px
    classDef api fill:#E0F8E0,stroke:#333,stroke-width:1px
    classDef workflow fill:#FFE8B0,stroke:#333,stroke-width:1px
    classDef engine fill:#FFE8B0,stroke:#333,stroke-width:1px
    classDef wrapper fill:#E8D0FF,stroke:#333,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
    classDef utils fill:#F8F8E0,stroke:#333,stroke-width:1px
```
