How saezlab/greta works
Project type and purpose
• GRETA is a Snakemake-based bioinformatics pipeline for inferring and benchmarking gene regulatory networks from single-cell multi-omics data.
• It leverages modular “methods,” “datasets,” and “databases” steps, runs them in containers, and submits jobs to an HPC cluster managed by SLURM.
File structure
• Top level:
• config/ – global pipeline configuration (config.yaml, prior_cats.json) and SLURM profiles/scripts
• workflow/ – the Snakemake workflow
– Snakefile – main orchestrator
– envs/ – Singularity definition (.def) files and resulting images (.sif)
– rules/ – rule subdirectories (anl, dbs, dts, mth, plt) each containing .smk rule files
– scripts/ – helper scripts organized in matching anl, dbs, dts, mth, plt subfolders
• docs/ – markdown guides for adding methods, datasets, databases
• LICENSE, README.md – project metadata
README for additional insights
• Confirms use of Snakemake, Singularity, and SLURM
• Shows typical snakemake --profile config/slurm/ target commands
• Describes modular steps (dts → preprocessing, mth → inference, anl → analysis, plt → plotting)
Main components
a. Config module
– config.yaml & prior_cats.json drive which datasets, methods, and databases to run
– SLURM profile scripts (slurm-submit.py, slurm-jobscript.sh, etc.)
b. Workflow engine (Snakemake)
– Reads Snakefile + config
– Resolves DAG of jobs
– Uses envs/ to create or pull containers
– Submits each rule as a SLURM job via the SLURM wrapper
c. Environment definitions
– Singularity .def files in workflow/envs/
– Build containers (.sif) per method/dataset
d. Rule modules
– Five domains: anl (analyses), dbs (databases), dts (datasets), mth (methods), plt (plots)
– Each rule invokes a helper script in workflow/scripts/…
e. Helper scripts
– Python/R/awk scripts that implement the actual data transformations
f. HPC cluster & scheduler
– SLURM submission and monitoring
– Sidecar scripts handle job status, submission, resource parameters
g. Data flow
– Input raw data (e.g., downloaded single-cell data) → processed data files (H5MU, CSVs) → aggregated metrics and plots
– Final outputs: summary CSVs, plots, benchmark tables
Relationships and interactions
• Config → Snakefile
• Snakefile → Rules (dts, mth, dbs, anl, plt)
• Each Rule → invokes a Container (envs/.sif) + Script → reads inputs → writes outputs
• Snakemake → SLURM wrapper → SLURM scheduler → allocates compute nodes and runs containerized jobs
• Outputs feed downstream rules until final reports/plots
Architectural patterns and design principles
• Modular: separate rules/domains with clear folder boundaries
• Reproducible: containerized environments, version-controlled configs
• Scalable: parallel execution via Snakemake + SLURM on HPC
• Extensible: new methods, datasets, or databases added by dropping .smk and helper scripts
Detail without over-thinking
• Break the pipeline into as many logical pieces as possible: config files, scheduler scripts, containers, rule sets, scripts, data stores
• File paths or folder names as sublabels under each component