# dbespiatykh/rdscan architecture

> Architecture diagram of the GitHub repository dbespiatykh/rdscan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbespiatykh/rdscan
- Repository: https://github.com/dbespiatykh/rdscan

## Overview

##### Project Overview and Main Components
- Project Type & Purpose: RDscan is a bioinformatics pipeline that uses Snakemake to analyze sequencing data for detecting genomic deletions (regions of difference) in Mycobacterium tuberculosis complex. Its primary purpose is to automate the analysis workflow – from raw sequencing inputs through mapping, discovery, and result reporting.
- Main Components:
- Input Data: Raw reads (FASTQ files under the .test/reads directory) and sample configuration files (.test/config/samples.tsv and config samples.tsv).
- Configuration Management: “config” directory (with its README and config.yml) and additional test configuration under “.test/config”. These define thresholds (e.g., for conditional formatting in RD_known.xlsx) and pipeline parameters.
- Scripts & Tools: Custom scripts are located in “workflow/scripts” (such as annotate_novel.py, concatenate_bed.py, makeTables.R, proportions.py) handling specific analyses or formatting tasks.
- Environment & Dependencies: The “environment.yml” and the files under “workflow/envs” specify conda-based environments and dependencies ensuring reproducible computational environments.
- External Resources: The “resources” directory contains reference data (like IS6110.bed, RD.bed, RvD1_TbD1.fa) needed for analysis.
- Testing & CI: The “.github” and “.test” directories indicate continuous integration workflows (GitHub Actions) and testing datasets to validate pipeline performance.

##### Relationships and Data Flow
- The configuration files (from both “config” and “.test/config”) feed into the Snakemake workflow to specify parameters for each rule.
- The Snakefile orchestrates multiple rule files (mapping, novel discovery, etc.) that process the raw data. These rules call custom scripts for tasks like annotation and generating summary tables.
- Throughout the workflow, external resources and reference files are accessed, and the environment is managed by conda (ensuring all dependencies are available).
- Finally, the output is generated (results tables such as RD_putative.tsv, RD_known.tsv, etc.) and possibly transformed (e.g., conditional formatting in the XLSX file as shown in the README).

##### Architectural Patterns and Design Principles
- Modular Design: The project is clearly organized into modular components (configuration, workflow rules, scripts, and resources) making it easier to extend or modify parts of the workflow.
- Separation of Concerns: Input data handling, processing logic, and output generation are distinct. The rules in the Snakemake workflow specifically focus on different analysis steps.
- Reproducibility: Use of conda environments (detailed in environment.yml and various env YAML files) ensures that the system is reproducible across environments.
- Workflow Orchestration: The use of Snakemake indicates a dataflow paradigm with clear dependencies between steps managed by the Snakefile.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Data
    subgraph "Input Data"
        Raw["Raw Sequencing Data (.fastq)"]:::input
    end

    %% Configuration Management
    subgraph "Configuration Management"
        MainConfig["Main Config (config/config.yml, config/samples.tsv)"]:::config
        TestConfig["Test Config (.test/config/config.yml, .test/config/samples.tsv)"]:::config
    end

    %% Workflow Engine
    subgraph "Workflow Engine"
        Snakefile["Snakefile (Snakemake)"]:::workflow
        RuleFiles["Rule Files (common.smk, mapping.smk, known_rds.smk, novel_discovery.smk, reference.smk)"]:::workflow
    end

    %% Custom Scripts & Tools
    subgraph "Custom Scripts & Tools"
        Script1["annotate_novel.py"]:::script
        Script2["concatenate_bed.py"]:::script
        Script3["makeTables.R"]:::script
        Script4["proportions.py"]:::script
    end

    %% Environment & Dependencies
    subgraph "Environment & Dependencies"
        Env1["environment.yml"]:::env
        EnvDir["Envs (calculations.yaml, entrez.yaml, gatk4.yaml, renv.yaml)"]:::env
    end

    %% External Resources
    subgraph "External Resources"
        Ref["Reference Data (IS6110.bed, RD.bed, RvD1_TbD1.fa)"]:::external
        TestRes["Test Resources (.test/resources)"]:::external
    end

    %% Testing & Continuous Integration
    subgraph "Testing & CI"
        CI["CI Workflows (.github)"]:::testing
        TestData["Test Datasets (.test)"]:::testing
    end

    %% Output / Results
    subgraph "Output"
        Output["Pipeline Results (Results tables, Reports)"]:::output
    end

    %% Data Flow Connections
    Raw -->|"provides_raw"| MainConfig
    Raw -->|"provides_raw"| Snakefile
    MainConfig -->|"passes_params"| Snakefile
    TestConfig -->|"passes_params"| Snakefile
    Snakefile -->|"orchestrates"| RuleFiles
    RuleFiles -->|"executes_script"| Script1
    RuleFiles -->|"executes_script"| Script2
    RuleFiles -->|"executes_script"| Script3
    RuleFiles -->|"executes_script"| Script4
    Env1 -->|"configures"| Snakefile
    EnvDir -->|"configures"| Snakefile
    RuleFiles -->|"accesses"| Ref
    RuleFiles -->|"produces"| Output
    CI -->|"triggers"| Snakefile
    TestData -->|"supplies_test"| Raw
    TestData -->|"validates"| RuleFiles
    Ref --- TestRes

    %% Click Events for Input Data
    click Raw "https://github.com/dbespiatykh/rdscan/tree/master/.test/reads"

    %% Click Events for Configuration Management
    click MainConfig "https://github.com/dbespiatykh/rdscan/blob/master/config/config.yml"
    click MainConfig "https://github.com/dbespiatykh/rdscan/blob/master/config/samples.tsv"
    click TestConfig "https://github.com/dbespiatykh/rdscan/blob/master/.test/config/config.yml"
    click TestConfig "https://github.com/dbespiatykh/rdscan/blob/master/.test/config/samples.tsv"

    %% Click Events for Workflow Engine
    click Snakefile "https://github.com/dbespiatykh/rdscan/tree/master/workflow/Snakefile"
    click RuleFiles "https://github.com/dbespiatykh/rdscan/tree/master/workflow/rules"

    %% Click Events for Custom Scripts & Tools
    click Script1 "https://github.com/dbespiatykh/rdscan/blob/master/workflow/scripts/annotate_novel.py"
    click Script2 "https://github.com/dbespiatykh/rdscan/blob/master/workflow/scripts/concatenate_bed.py"
    click Script3 "https://github.com/dbespiatykh/rdscan/blob/master/workflow/scripts/makeTables.R"
    click Script4 "https://github.com/dbespiatykh/rdscan/blob/master/workflow/scripts/proportions.py"

    %% Click Events for Environment & Dependencies
    click Env1 "https://github.com/dbespiatykh/rdscan/blob/master/environment.yml"
    click EnvDir "https://github.com/dbespiatykh/rdscan/blob/master/workflow/envs/calculations.yaml"
    click EnvDir "https://github.com/dbespiatykh/rdscan/blob/master/workflow/envs/entrez.yaml"
    click EnvDir "https://github.com/dbespiatykh/rdscan/blob/master/workflow/envs/gatk4.yaml"
    click EnvDir "https://github.com/dbespiatykh/rdscan/blob/master/workflow/envs/renv.yaml"

    %% Click Events for External Resources
    click Ref "https://github.com/dbespiatykh/rdscan/blob/master/resources/IS6110.bed"
    click Ref "https://github.com/dbespiatykh/rdscan/blob/master/resources/RD.bed"
    click Ref "https://github.com/dbespiatykh/rdscan/blob/master/resources/RvD1_TbD1.fa"
    click TestRes "https://github.com/dbespiatykh/rdscan/tree/master/.test/resources"

    %% Click Events for Testing & CI
    click CI "https://github.com/dbespiatykh/rdscan/blob/master/.github"
    click TestData "https://github.com/dbespiatykh/rdscan/blob/master/.test"

    %% Class Definitions
    classDef input fill:#ffcc00,stroke:#000,stroke-width:1px;
    classDef config fill:#66ccff,stroke:#000,stroke-width:1px;
    classDef workflow fill:#8fbc8f,stroke:#000,stroke-width:1px;
    classDef script fill:#daa520,stroke:#000,stroke-width:1px;
    classDef env fill:#d3d3d3,stroke:#000,stroke-width:1px;
    classDef external fill:#87ceeb,stroke:#000,stroke-width:1px;
    classDef testing fill:#9370db,stroke:#000,stroke-width:1px;
    classDef output fill:#ffa07a,stroke:#000,stroke-width:1px;

    %% Assign Classes
    class Raw input;
    class MainConfig,TestConfig config;
    class Snakefile,RuleFiles workflow;
    class Script1,Script2,Script3,Script4 script;
    class Env1,EnvDir env;
    class Ref,TestRes external;
    class CI,TestData testing;
    class Output output;
```
