# sdparekh/zumis architecture

> Architecture diagram of the GitHub repository sdparekh/zumis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sdparekh/zumis
- Repository: https://github.com/sdparekh/zumis

## Overview

##### Project Type & Purpose
- zUMIs is an open-source, command-line bioinformatics pipeline for single-cell RNA-seq UMI processing. It orchestrates a sequence of filtering, demultiplexing, mapping, UMI/barcode correction, counting, and downstream export (including loom) steps via a master Bash script, multi-language helper scripts, and external tools (STAR, samtools, BBMap, velocyto).

##### File-Structure Analysis & Architectural Patterns
- Top-Level Orchestrator:
- zUMIs.sh (Bash) – entry point that parses zUMIs.yaml, sets up the Conda environment (-c), and dispatches jobs.
- zUMIs.yaml – user-editable config with all pipeline parameters (YAML).
- Core Modules (scripts triggered in sequence):
- Preprocessing: splitfq.sh, fqfilter_v2.pl (Perl), merge_demultiplexed_fastq.R (R), listPrefix.sh (shell)
- Demultiplex & UMI/BC handling: misc/demultiplex_BC.{pl,py}, zUMIs-BCdetection.R, correct_UBtag.{pl,py}, correct_BCtag.pl
- Mapping: zUMIs-mapping.R, zUMIs-bbmap.R orchestrate external mappers (STAR, BBMap) and samtools
- Counting & Stats: zUMIs-stats2.R, runfeatureCountFUN.R, featureCounts.R, statsFUN.R
- Post-processing & Output: zUMIs-dge2.R (matrix assembly), rds2loom.R (loom conversion), runVelocyto.R
- Support Libraries & Helpers:
- Approx.pm, distilReads.pm (Perl modules), misc/fcountsLib* (featureCounts bindings)
- checkyaml.R, readYaml4fqfilter.R for config validation/parsing
- External Tools & Dependencies: bundled Miniconda environment parts (zUMIs-miniconda.part[a-h]), STAR index usage, R, Python, Perl, BBMap, velocyto.

1) Input Layer: FastQ files, STAR genome index, GTF annotation, zUMIs.yaml
2) Orchestration Layer: zUMIs.sh + environment manager (Conda)
3) Preprocessing Module
4) Demultiplexing & UMI/BC Correction Module
5) Mapping Module
6) Counting & Statistics Module
7) Post-processing & Export Module
8) Output Layer: BAMs, count matrices (.rds/.txt), stats plots, loom files
- Input → Orchestrator → Config parsing → Preprocessing → Demultiplexing → Correction → Mapping → Counting → DGE matrix creation → Optional RNA-velocity → Loom conversion → Outputs
c. Architectural Patterns & Principles:
- Pipeline pattern (linear, modular stages)
- Modular multi-language scripts (Bash top-level, R for stats, Perl/Python for fast processing)
- Config-driven (YAML) for reproducibility & extensibility
- Parallelization at key steps (filtering, mapping, counting)
d. Key Technologies / Libraries:
- Bash, Conda, YAML, R (Rscript), Perl, Python (pysam)
- External: STAR, samtools, BBMap, velocyto, Rsubread::featureCounts, loomR

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input Layer
    subgraph "Input Layer"
        direction TB
        FastQ["FastQ Files"]:::data
        STARIndex["STAR Genome Index"]:::data
        GTF["GTF Annotation"]:::data
        ConfigYAML["zUMIs.yaml"]:::config
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        Orchestrator["zUMIs.sh"]:::orchestration
    end

    %% Preprocessing Module
    subgraph "Preprocessing Module"
        direction TB
        SplitFastq["splitfq.sh"]:::helper
        FilterFastq["fqfilter_v2.pl"]:::helper
        ListPrefix["listPrefix.sh"]:::helper
        MergeFastq["merge_demultiplexed_fastq.R"]:::rscript
    end

    %% Demultiplexing & UMI/BC Correction Module
    subgraph "Demultiplex & UMI/BC Correction"
        direction TB
        DemuxPL["misc/demultiplex_BC.pl"]:::helper
        DemuxPY["misc/demultiplex_BC.py"]:::helper
        BCdetect["zUMIs-BCdetection.R"]:::rscript
        CorrUBPL["correct_UBtag.pl"]:::helper
        CorrUBPY["correct_UBtag.py"]:::helper
        CorrBC["correct_BCtag.pl"]:::helper
    end

    %% Mapping Module
    subgraph "Mapping Module"
        direction TB
        MapR["zUMIs-mapping.R"]:::rscript
        BBMapR["zUMIs-bbmap.R"]:::rscript
        MergeBAM["mergeBAM.sh"]:::helper
        STAR["STAR"]:::external
        Samtools["samtools"]:::external
        BBMap["BBMap"]:::external
    end

    %% Counting & Statistics Module
    subgraph "Counting & Statistics Module"
        direction TB
        RunFC["runfeatureCountFUN.R"]:::rscript
        FCwrap["misc/featureCounts.R"]:::rscript
        Stats2["zUMIs-stats2.R"]:::rscript
        StatsFUN["statsFUN.R"]:::rscript
        FeatureCounts["Rsubread::featureCounts"]:::external
    end

    %% Post-processing & Export Module
    subgraph "Post-processing & Export Module"
        direction TB
        DGE["zUMIs-dge2.R"]:::rscript
        RDS2Loom["misc/rds2loom.R"]:::rscript
        Velocyto["runVelocyto.R"]:::rscript
    end

    %% Output Layer
    subgraph "Output Layer"
        direction TB
        BAMs["BAM Files"]:::data
        Counts["Count Matrices (.rds/.txt)"]:::data
        StatsPlots["Stats Plots"]:::data
        Loom["Loom Files"]:::data
    end

    %% Data Flow
    FastQ --> Orchestrator
    STARIndex --> Orchestrator
    GTF --> Orchestrator
    ConfigYAML --> Orchestrator

    Orchestrator --> SplitFastq
    SplitFastq --> FilterFastq
    FilterFastq --> ListPrefix
    ListPrefix --> MergeFastq

    MergeFastq --> DemuxPL
    DemuxPL --> BCdetect
    DemuxPY --> BCdetect
    BCdetect --> CorrUBPL
    BCdetect --> CorrUBPY
    CorrUBPL --> CorrBC
    CorrUBPY --> CorrBC
    CorrBC --> MapR

    MapR --> STAR
    MapR --> Samtools
    BBMapR --> BBMap
    STAR --> MergeBAM
    Samtools --> MergeBAM
    BBMap --> MergeBAM
    MergeBAM --> RunFC

    RunFC --> FCwrap
    FCwrap --> FeatureCounts
    FeatureCounts --> Stats2
    Stats2 --> StatsFUN
    StatsFUN --> DGE

    DGE --> RDS2Loom
    RDS2Loom --> Loom
    DGE --> Velocyto
    Velocyto --> Loom

    MergeBAM --> BAMs
    DGE --> Counts
    StatsFUN --> StatsPlots

    %% Click Events
    click Orchestrator "https://github.com/sdparekh/zumis/blob/main/zUMIs.sh"
    click ConfigYAML "https://github.com/sdparekh/zumis/blob/main/zUMIs.yaml"
    click SplitFastq "https://github.com/sdparekh/zumis/blob/main/splitfq.sh"
    click FilterFastq "https://github.com/sdparekh/zumis/blob/main/fqfilter_v2.pl"
    click ListPrefix "https://github.com/sdparekh/zumis/blob/main/listPrefix.sh"
    click MergeFastq "https://github.com/sdparekh/zumis/blob/main/misc/merge_demultiplexed_fastq.R"
    click DemuxPL "https://github.com/sdparekh/zumis/blob/main/misc/demultiplex_BC.pl"
    click DemuxPY "https://github.com/sdparekh/zumis/blob/main/misc/demultiplex_BC.py"
    click BCdetect "https://github.com/sdparekh/zumis/blob/main/zUMIs-BCdetection.R"
    click CorrUBPL "https://github.com/sdparekh/zumis/blob/main/correct_UBtag.pl"
    click CorrUBPY "https://github.com/sdparekh/zumis/blob/main/correct_UBtag.py"
    click CorrBC "https://github.com/sdparekh/zumis/blob/main/correct_BCtag.pl"
    click MapR "https://github.com/sdparekh/zumis/blob/main/zUMIs-mapping.R"
    click BBMapR "https://github.com/sdparekh/zumis/blob/main/zUMIs-bbmap.R"
    click MergeBAM "https://github.com/sdparekh/zumis/blob/main/mergeBAM.sh"
    click RunFC "https://github.com/sdparekh/zumis/blob/main/runfeatureCountFUN.R"
    click FCwrap "https://github.com/sdparekh/zumis/blob/main/misc/featureCounts.R"
    click Stats2 "https://github.com/sdparekh/zumis/blob/main/zUMIs-stats2.R"
    click StatsFUN "https://github.com/sdparekh/zumis/blob/main/statsFUN.R"
    click DGE "https://github.com/sdparekh/zumis/blob/main/zUMIs-dge2.R"
    click RDS2Loom "https://github.com/sdparekh/zumis/blob/main/misc/rds2loom.R"
    click Velocyto "https://github.com/sdparekh/zumis/blob/main/runVelocyto.R"

    %% Styles
    classDef orchestration fill:#a2f5a2,stroke:#000;
    classDef config fill:#a2f5a2,stroke:#000;
    classDef helper fill:#fcf1a0,stroke:#000;
    classDef rscript fill:#a0c4fc,stroke:#000;
    classDef external fill:#fdba74,stroke:#000;
    classDef data fill:#d3d3d3,stroke:#000;
```
