# hazyresearch/hyena-dna architecture

> Architecture diagram of the GitHub repository hazyresearch/hyena-dna, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hazyresearch/hyena-dna
- Repository: https://github.com/hazyresearch/hyena-dna

## Overview

This project is a machine learning research and development repository focused on genomic sequence modeling via HyenaDNA. In other words, it implements a long-range foundation model for genomic data (DNA sequences at single nucleotide resolution) that can be pretrained and then fine-tuned on various downstream tasks such as genomic benchmarks, chromatin profiling, species classification, and even in-context learning experiments. It is built with PyTorch Lightning, uses Hydra for configuration management, integrates with Docker for reproducible environments, and includes custom CUDA/C++ operations (e.g., FFT convolution) through its “csrc” directory as well as a submodule for Flash Attention.

##### Main Components of the System
- Core Training Pipeline – Representing the entry point (train.py) and the high-level orchestration via PyTorch Lightning. This includes the training loop, experiment management (with Hydra configuration), and custom callbacks (e.g., sequence length warmup).
- Model Architecture – Detail the backbone model that is defined in “src/models/sequence” (which includes modules like block, hyena, mha, etc.), along with auxiliary model components (e.g., decoders for downstream tasks, classification heads). Also note the usage of specialized layers defined in “src/models/nn” and the custom FFT operations in “src/ops”.
- Data Loading & Preprocessing – Include the dataloaders and dataset classes in “src/dataloaders” (with specialized dataset implementations such as genomic_bench_dataset, nucleotide_transformer_dataset, etc.). Also mention the external data files expected in the “data” directory.
- Configuration & Experiment Management – Illustrate how the project uses Hydra with extensive configuration files (located in “configs/” directories) to control everything from model hyperparameters to training tasks. This is essential to support multiple experiments (e.g., pretraining, finetuning, in-context learning).
- Custom C++/CUDA Operations – Highlight the “csrc” module which implements the performance-critical FFT convolution and related operations. These custom operations integrate with Python and are used by the model layers for efficiency.
- Evaluation & Inference – Represent the “evals” directory that includes scripts for inference (e.g., hg38_inference.py, soft_prompting_genomics.py) and also the “huggingface.py” and “standalone_hyenadna.py” for alternative inference pathways.
- Docker and Submodules – Show that Docker is used to package the entire environment (using Dockerfile) and note that external submodules (like flash-attention) are integrated for additional functionality.

##### Relationships and Interactions
- The configuration system (Hydra) feeds settings to the training pipeline—influencing model parameters (such as d_model, n_layer, l_max), dataloader settings (batch size, maximum sequence length), and optimizer/hyperparameter settings.
- The training script (train.py) is the central coordinator that invokes the dataloaders (which load data from “data” directories or external sources), instantiates the backbone model (from “src/models/…”), and launches experiments via PyTorch Lightning.
- Custom CUDA extensions and C++ operations (from “csrc”) are built during installation or via Docker and are used by the model layers to provide optimized numerical operations.
- Evaluation scripts and inference modules interact with the trained model (and optionally pretrained weights from HuggingFace) to serve as downstream tasks or generate embeddings/logits.
- Docker images and Dockerfile guarantee a reproducible environment for development and execution, and are used either directly or via containerized experiments.
- There is a clear separation of concerns: configuration management, model definition, data handling, and training orchestration are structured in separate directories.

##### Architectural Patterns and Design Principles
- The project employs a modular design that decouples configuration, model definition, training routines, and evaluation. Hydra is used to make experiments easily switchable via configuration overlays.
- It uses the “separation of concerns” principle to separate the data pipeline (dataloaders and datasets) from the model logic (“src/models”) and training orchestration (train.py plus task definitions in “src/tasks”).
- Custom performance-critical components (C++/CUDA in “csrc”) are kept distinct so that they can be maintained and updated independently.
- The system is designed to be extendable: new datasets and tasks can be added with minimal changes by following the configuration structure in “configs” and the class registration mechanisms in “src/utils/registry.py”.

##### Technologies, Frameworks, and Libraries
- PyTorch Lightning for managing training loops and distributed training.
- Hydra for configuration management, with numerous YAML config files for experiments, datasets, models, pipelines, optimizers, etc.
- Docker to encapsulate dependencies and runtime requirements.
- Custom C++/CUDA code (in “csrc”) for FFT convolution and Flash Attention operations.
- External repositories/submodules such as Flash-Attention, and links to HuggingFace pretrained model weights.
- Datasets and evaluation scripts designed to interface with genomics-specific file formats (.fasta, .bed) for genomic data.

##### Specific Elements
- Configuration Manager (Hydra with configs)
- Training Pipeline / PyTorch Lightning (train.py and tasks)
- Model Architecture (backbone + decoder) with submodules from “src/models/sequence” and “src/models/nn”
- Data Loader & Dataset Modules (in “src/dataloaders”)
- Custom Ops (C++/CUDA in “csrc”) integrated with model layers
- Evaluation Components (scripts in “evals/” and standalone inference modules)
- Docker Environment and Submodule integration
- How configs drive both training and data handling
- Flow of data from raw input files (e.g., fasta) into dataloaders and then into the model
- Model outputs being passed to evaluation modules
- Core training elements (e.g., model and training loop) versus
- Auxiliary components (e.g., Docker setup, custom ops)
- Annotation notes to mention key external references (such as HuggingFace integration, Docker image usage, and submodule updates).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration & Experiment Management"
        config["Hydra Config (configs)"]
    end

    subgraph "Training & Modeling"
        pipeline["Training Pipeline (train.py)"]
        dataloader["Data Loader (src/dataloaders)"]
        subgraph "Model Architecture"
            backbone["Backbone Model (src/models/sequence)"]
            nnLayers["Specialized NN Layers (src/models/nn)"]
        end
        customOps["Custom C++/CUDA Ops (csrc)"]
    end

    subgraph "Evaluation & Inference"
        evaluation["Evaluation Module (evals, huggingface.py, standalone_hyenadna.py)"]
    end

    subgraph "Deployment & External"
        docker["Docker Environment (Dockerfile)"]
        flash["Flash-Attention Submodule (flash-attention)"]
    end

    config -->|"configures"| pipeline
    config -->|"configures"| dataloader
    pipeline -->|"invokes"| dataloader
    pipeline -->|"trains"| backbone
    pipeline -->|"trains"| nnLayers
    backbone -->|"utilizes"| customOps
    nnLayers -->|"utilizes"| customOps
    pipeline -->|"outputs"| evaluation
    docker -->|"containers"| pipeline
    docker -->|"containers"| dataloader
    docker -->|"containers"| backbone
    docker -->|"containers"| nnLayers
    flash -->|"integrates"| customOps

    class config config
    class pipeline,dataloader,backbone,nnLayers training
    class customOps custom
    class evaluation evaluation
    class docker docker
    class flash external

    classDef config fill:#D2B4DE,stroke:#6C3483,stroke-width:2px
    classDef training fill:#AED6F1,stroke:#2874A6,stroke-width:2px
    classDef custom fill:#F5B7B1,stroke:#C0392B,stroke-width:2px
    classDef evaluation fill:#F9E79F,stroke:#B7950B,stroke-width:2px
    classDef docker fill:#A9DFBF,stroke:#229954,stroke-width:2px
    classDef external fill:#CCD1D1,stroke:#566573,stroke-width:2px

    click pipeline "https://github.com/hazyresearch/hyena-dna/blob/main/train.py"
    click backbone "https://github.com/hazyresearch/hyena-dna/tree/main/src/models/sequence"
    click nnLayers "https://github.com/hazyresearch/hyena-dna/tree/main/src/models/nn"
    click dataloader "https://github.com/hazyresearch/hyena-dna/tree/main/src/dataloaders"
    click config "https://github.com/hazyresearch/hyena-dna/tree/main/configs"
    click customOps "https://github.com/hazyresearch/hyena-dna/tree/main/csrc"
    click evaluation "https://github.com/hazyresearch/hyena-dna/tree/main/evals"
    click docker "https://github.com/hazyresearch/hyena-dna/tree/main/Dockerfile"
    click flash "https://github.com/hazyresearch/hyena-dna/tree/main/flash-attention"
```
