# zhanglab-aim/dyna architecture

> Architecture diagram of the GitHub repository zhanglab-aim/dyna, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhanglab-aim/dyna
- Repository: https://github.com/zhanglab-aim/dyna

## Overview

##### Project Type and Purpose
- This is a research-oriented machine-learning pipeline (not a web app or microservice) focused on disease-specific variant effect prediction (VEP).
- DYNA fine-tunes pre-trained genomic foundation models (protein and DNA language models) via a Siamese neural network to classify pathogenic vs. benign variants in coding and non-coding contexts.

##### Main Components
a. External Data Sources
- ClinVar (CM, ARM, splicing datasets)
- Multiplex assay (exon recognition data)
- Genome annotation files (e.g., GFF3)
- Pfam domain definitions
b. Data Preprocessing (dyna_process_data/)
- Jupyter notebooks that ingest raw variant and sequence data, annotate sequences (coding vs. non-coding), translate DNA→protein, extract features.
c. Core Model Pipeline (dyna/)
- Embedding generation using foundation models (ESM-1b, ESM, etc.)
- Siamese network architecture (two shared-weight branches) implemented in Python scripts and notebooks.
- Fine-tuning routines for coding and non-coding VEP tasks.
d. Experimentation & Visualization
- Minimal example notebooks (minimal_example.ipynb, minimal_example_non_coding.ipynb) demonstrating usage.
e. Orchestration & Deployment
- SLURM batch scripts (scripts/test_gpu.sh) to submit training jobs on GPU clusters.
- Environment specifications via dyna.yml and requirements.txt.
f. Outputs
- Trained model checkpoints (linked externally under dyna_model/)
- Performance evaluations and figures.

##### Relationships and Data Flow
- The SLURM script encapsulates the “dyna/” training commands and schedules GPU resources.
- Notebooks tap into both preprocessing outputs and model code for demos and analyses.

##### Architectural Patterns & Design Principles
- Modular separation: Data ingestion, preprocessing, model training, evaluation, and visualization live in distinct directories.
- Reusable Siamese-network module that wraps arbitrary foundation model embeddings.
- Experiment scripts and notebooks are decoupled from core training logic.
- HPC integration via lightweight batch scripts.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    ExternalData["External Data Sources"]:::dataComp

    %% Preprocessing Module
    subgraph "Preprocessing Module (dyna_process_data)" 
        direction TB
        PC1["preprocess_clinvar_new.ipynb"]:::computeComp
        PC2["process_clinvar_arm_new.ipynb"]:::computeComp
        PC3["process_clinvar_cm_new.ipynb"]:::computeComp
        PC4["snv_data_preprocessing.ipynb"]:::computeComp
    end

    %% Embedding & Siamese Training Module
    subgraph "Core Model Pipeline (dyna)" 
        direction TB
        ESM1["VEP_esm1b_PLLR_ar.ipynb"]:::computeComp
        ESM2["VEP_esm_PLLR.ipynb"]:::computeComp
        ESM3["VEP_esm_PLLR_cm_clinvar.ipynb"]:::computeComp
        ESM4["VEP_esm_protein_siamese.ipynb"]:::computeComp
        PY1["VEP_GPN_siamese_contrastive_Clinvar_MFASS.py"]:::computeComp
        PY2["VEP_nt_siamese_nt.py"]:::computeComp
    end

    %% Experimentation & Visualization
    subgraph "Experimentation & Visualization" 
        direction TB
        ME1["minimal_example.ipynb"]:::computeComp
        ME2["minimal_example_non_coding.ipynb"]:::computeComp
        FIGS["figures/"]:::outputComp
    end

    %% Orchestration & Environment
    subgraph "Orchestration & Configuration"
        direction TB
        SLURM["scripts/test_gpu.sh"]:::orchestrationComp
        ENV1["dyna.yml"]:::configComp
        ENV2["requirements.txt"]:::configComp
    end

    %% Model Storage
    ModelStorage["Model Storage (checkpoints)"]:::outputComp

    %% Workflow Arrows
    ExternalData --> PC1
    ExternalData --> PC2
    ExternalData --> PC3
    ExternalData --> PC4

    PC1 --> ESM1
    PC2 --> ESM2
    PC3 --> ESM3
    PC4 --> ESM4

    ESM1 --> PY1
    ESM2 --> PY1
    ESM3 --> PY1
    ESM4 --> PY2

    PY1 --> ME1
    PY2 --> ME2

    ME1 --> FIGS
    ME2 --> FIGS

    SLURM --> PY1
    SLURM --> PY2

    ENV1 --> SLURM
    ENV2 --> SLURM

    PY1 --> ModelStorage
    PY2 --> ModelStorage

    %% Click Events
    click PC1 "https://github.com/zhanglab-aim/dyna/blob/main/dyna_process_data/preprocess_clinvar_new.ipynb"
    click PC2 "https://github.com/zhanglab-aim/dyna/blob/main/dyna_process_data/process_clinvar_arm_new.ipynb"
    click PC3 "https://github.com/zhanglab-aim/dyna/blob/main/dyna_process_data/process_clinvar_cm_new.ipynb"
    click PC4 "https://github.com/zhanglab-aim/dyna/blob/main/dyna_process_data/snv_data_preprocessing.ipynb"
    click ESM1 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_esm1b_PLLR_ar.ipynb"
    click ESM2 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_esm_PLLR.ipynb"
    click ESM3 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_esm_PLLR_cm_clinvar.ipynb"
    click ESM4 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_esm_protein_siamese.ipynb"
    click PY1 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_GPN_siamese_contrastive_Clinvar_MFASS.py"
    click PY2 "https://github.com/zhanglab-aim/dyna/blob/main/dyna/VEP_nt_siamese_nt.py"
    click ME1 "https://github.com/zhanglab-aim/dyna/blob/main/minimal_example.ipynb"
    click ME2 "https://github.com/zhanglab-aim/dyna/blob/main/minimal_example_non_coding.ipynb"
    click FIGS "https://github.com/zhanglab-aim/dyna/tree/main/figures/"
    click SLURM "https://github.com/zhanglab-aim/dyna/blob/main/scripts/test_gpu.sh"
    click ENV1 "https://github.com/zhanglab-aim/dyna/blob/main/dyna.yml"
    click ENV2 "https://github.com/zhanglab-aim/dyna/blob/main/requirements.txt"

    %% Styles
    classDef dataComp fill:#8FBC8F,stroke:#333,stroke-width:1px
    classDef computeComp fill:#87CEEB,stroke:#333,stroke-width:1px
    classDef orchestrationComp fill:#FFA500,stroke:#333,stroke-width:1px
    classDef outputComp fill:#9370DB,stroke:#333,stroke-width:1px
    classDef configComp fill:#D3D3D3,stroke:#333,stroke-width:1px
```
