# saezlab/greta architecture

> Architecture diagram of the GitHub repository saezlab/greta, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saezlab/greta
- Repository: https://github.com/saezlab/greta

## Overview

##### Project type and purpose
- GRETA is a Snakemake-based bioinformatics pipeline for inferring and benchmarking gene regulatory networks from single-cell multi-omics data.
- It leverages modular “methods,” “datasets,” and “databases” steps, runs them in containers, and submits jobs to an HPC cluster managed by SLURM.

##### File structure
- Top level:
- config/ – global pipeline configuration (config.yaml, prior_cats.json) and SLURM profiles/scripts
- workflow/ – the Snakemake workflow
– Snakefile – main orchestrator
– envs/ – Singularity definition (.def) files and resulting images (.sif)
– rules/ – rule subdirectories (anl, dbs, dts, mth, plt) each containing .smk rule files
– scripts/ – helper scripts organized in matching anl, dbs, dts, mth, plt subfolders
- docs/ – markdown guides for adding methods, datasets, databases
- LICENSE, README.md – project metadata

##### README for additional insights
- Confirms use of Snakemake, Singularity, and SLURM
- Shows typical `snakemake --profile config/slurm/ target` commands
- Describes modular steps (dts → preprocessing, mth → inference, anl → analysis, plt → plotting)

##### Main components
a. Config module
– config.yaml & prior_cats.json drive which datasets, methods, and databases to run
– SLURM profile scripts (slurm-submit.py, slurm-jobscript.sh, etc.)
b. Workflow engine (Snakemake)
– Reads Snakefile + config
– Resolves DAG of jobs
– Uses envs/ to create or pull containers
– Submits each rule as a SLURM job via the SLURM wrapper
c. Environment definitions
– Singularity .def files in workflow/envs/
– Build containers (.sif) per method/dataset
d. Rule modules
– Five domains: anl (analyses), dbs (databases), dts (datasets), mth (methods), plt (plots)
– Each rule invokes a helper script in workflow/scripts/…
e. Helper scripts
– Python/R/awk scripts that implement the actual data transformations
f. HPC cluster & scheduler
– SLURM submission and monitoring
– Sidecar scripts handle job status, submission, resource parameters
g. Data flow
– Input raw data (e.g., downloaded single-cell data) → processed data files (H5MU, CSVs) → aggregated metrics and plots
– Final outputs: summary CSVs, plots, benchmark tables

##### Relationships and interactions
- Config → Snakefile
- Snakefile → Rules (dts, mth, dbs, anl, plt)
- Each Rule → invokes a Container (envs/.sif) + Script → reads inputs → writes outputs
- Snakemake → SLURM wrapper → SLURM scheduler → allocates compute nodes and runs containerized jobs
- Outputs feed downstream rules until final reports/plots

##### Architectural patterns and design principles
- Modular: separate rules/domains with clear folder boundaries
- Reproducible: containerized environments, version-controlled configs
- Scalable: parallel execution via Snakemake + SLURM on HPC
- Extensible: new methods, datasets, or databases added by dropping .smk and helper scripts

##### Detail without over-thinking
- Break the pipeline into as many logical pieces as possible: config files, scheduler scripts, containers, rule sets, scripts, data stores
- File paths or folder names as sublabels under each component

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Config Layer
    subgraph "Config Layer"
        ConfigYAML["config/config.yaml & prior_cats.json"]:::config
        SLURMProfiles["SLURM Profiles (slurm/)"]:::config
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Snakefile["workflow/Snakefile"]:::orchestration
    end

    %% Environment Layer
    subgraph "Environment Layer"
        SingularityDefs["Singularity Definitions (.def/.yaml)"]:::execution
    end

    %% Processing Layer
    subgraph "Processing Layer"
        subgraph "Rule Modules"
            DtsRules["Datasets Rules (workflow/rules/dts/)"]:::processing
            MthRules["Methods Rules (workflow/rules/mth/)"]:::processing
            DbsRules["Databases Rules (workflow/rules/dbs/)"]:::processing
            AnlRules["Analysis Rules (workflow/rules/anl/)"]:::processing
            PltRules["Plotting Rules (workflow/rules/plt/)"]:::processing
        end
        subgraph "Helper Scripts"
            DtsScripts["Datasets Scripts (workflow/scripts/dts/)"]:::processing
            MthScripts["Methods Scripts (workflow/scripts/mth/)"]:::processing
            DbsScripts["Databases Scripts (workflow/scripts/dbs/)"]:::processing
            AnlScripts["Analysis Scripts (workflow/scripts/anl/)"]:::processing
            PltScripts["Plotting Scripts (workflow/scripts/plt/)"]:::processing
        end
    end

    %% Execution Layer
    subgraph "Execution Layer"
        SLURMWrapper["SLURM Wrapper Scripts"]:::execution
        HPC["HPC Cluster (SLURM Scheduler)"]:::external
    end

    %% Data Layer
    subgraph "Data Layer"
        RawData["Raw Data"]:::data
        Processed["Processed Data"]:::data
        FinalOutputs["Final Outputs (CSVs, Plots)"]:::data
    end

    %% Connections
    ConfigYAML -->|drives| Snakefile
    SLURMProfiles -->|profiles| Snakefile

    Snakefile -->|reads config| SingularityDefs
    Snakefile -->|orchestrates| DtsRules
    Snakefile -->|orchestrates| MthRules
    Snakefile -->|orchestrates| DbsRules
    Snakefile -->|orchestrates| AnlRules
    Snakefile -->|orchestrates| PltRules
    Snakefile -->|submits jobs via| SLURMWrapper

    SLURMWrapper -->|schedules on| HPC

    DtsRules -->|invokes| DtsScripts
    MthRules -->|invokes| MthScripts
    DbsRules -->|invokes| DbsScripts
    AnlRules -->|invokes| AnlScripts
    PltRules -->|invokes| PltScripts

    DtsScripts -->|reads| RawData
    RawData -->|transforms| Processed
    Processed -->|aggregates| FinalOutputs

    %% Click Events
    click ConfigYAML "https://github.com/saezlab/greta/blob/main/config/config.yaml"
    click SLURMProfiles "https://github.com/saezlab/greta/tree/main/config/slurm/"
    click Snakefile "https://github.com/saezlab/greta/tree/main/workflow/Snakefile"
    click SingularityDefs "https://github.com/saezlab/greta/tree/main/workflow/envs/"
    click DtsRules "https://github.com/saezlab/greta/tree/main/workflow/rules/dts/"
    click MthRules "https://github.com/saezlab/greta/tree/main/workflow/rules/mth/"
    click DbsRules "https://github.com/saezlab/greta/tree/main/workflow/rules/dbs/"
    click AnlRules "https://github.com/saezlab/greta/tree/main/workflow/rules/anl/"
    click PltRules "https://github.com/saezlab/greta/tree/main/workflow/rules/plt/"
    click DtsScripts "https://github.com/saezlab/greta/tree/main/workflow/scripts/dts/"
    click MthScripts "https://github.com/saezlab/greta/tree/main/workflow/scripts/mth/"
    click DbsScripts "https://github.com/saezlab/greta/tree/main/workflow/scripts/dbs/"
    click AnlScripts "https://github.com/saezlab/greta/tree/main/workflow/scripts/anl/"
    click PltScripts "https://github.com/saezlab/greta/tree/main/workflow/scripts/plt/"
    click SLURMWrapper "https://github.com/saezlab/greta/tree/main/config/slurm/"

    %% Styles
    classDef config fill:#ADD8E6,stroke:#000,color:#000
    classDef orchestration fill:#90EE90,stroke:#000,color:#000
    classDef execution fill:#FFA500,stroke:#000,color:#000
    classDef processing fill:#D3D3D3,stroke:#000,color:#000
    classDef data fill:#E0FFFF,stroke:#000,color:#000
    classDef external fill:#F0F8FF,stroke:#000,color:#000
```
