How zhanglab-aim/dyna works
Project Type and Purpose
• This is a research-oriented machine-learning pipeline (not a web app or microservice) focused on disease-specific variant effect prediction (VEP).
• DYNA fine-tunes pre-trained genomic foundation models (protein and DNA language models) via a Siamese neural network to classify pathogenic vs. benign variants in coding and non-coding contexts.
Main Components
a. External Data Sources
• ClinVar (CM, ARM, splicing datasets)
• Multiplex assay (exon recognition data)
• Genome annotation files (e.g., GFF3)
• Pfam domain definitions
b. Data Preprocessing (dyna_process_data/)
• Jupyter notebooks that ingest raw variant and sequence data, annotate sequences (coding vs. non-coding), translate DNA→protein, extract features.
c. Core Model Pipeline (dyna/)
• Embedding generation using foundation models (ESM-1b, ESM, etc.)
• Siamese network architecture (two shared-weight branches) implemented in Python scripts and notebooks.
• Fine-tuning routines for coding and non-coding VEP tasks.
d. Experimentation & Visualization
• Minimal example notebooks (minimal_example.ipynb, minimal_example_non_coding.ipynb) demonstrating usage.
e. Orchestration & Deployment
• SLURM batch scripts (scripts/test_gpu.sh) to submit training jobs on GPU clusters.
• Environment specifications via dyna.yml and requirements.txt.
f. Outputs
• Trained model checkpoints (linked externally under dyna_model/)
• Performance evaluations and figures.
Relationships and Data Flow
• The SLURM script encapsulates the “dyna/” training commands and schedules GPU resources.
• Notebooks tap into both preprocessing outputs and model code for demos and analyses.
Architectural Patterns & Design Principles
• Modular separation: Data ingestion, preprocessing, model training, evaluation, and visualization live in distinct directories.
• Reusable Siamese-network module that wraps arbitrary foundation model embeddings.
• Experiment scripts and notebooks are decoupled from core training logic.
• HPC integration via lightweight batch scripts.