How hazyresearch/hyena-dna works
This project is a machine learning research and development repository focused on genomic sequence modeling via HyenaDNA. In other words, it implements a long-range foundation model for genomic data (DNA sequences at single nucleotide resolution) that can be pretrained and then fine-tuned on various downstream tasks such as genomic benchmarks, chromatin profiling, species classification, and even in-context learning experiments. It is built with PyTorch Lightning, uses Hydra for configuration management, integrates with Docker for reproducible environments, and includes custom CUDA/C++ operations (e.g., FFT convolution) through its “csrc” directory as well as a submodule for Flash Attention.
Main Components of the System
• Core Training Pipeline – Representing the entry point (train.py) and the high-level orchestration via PyTorch Lightning. This includes the training loop, experiment management (with Hydra configuration), and custom callbacks (e.g., sequence length warmup).
• Model Architecture – Detail the backbone model that is defined in “src/models/sequence” (which includes modules like block, hyena, mha, etc.), along with auxiliary model components (e.g., decoders for downstream tasks, classification heads). Also note the usage of specialized layers defined in “src/models/nn” and the custom FFT operations in “src/ops”.
• Data Loading & Preprocessing – Include the dataloaders and dataset classes in “src/dataloaders” (with specialized dataset implementations such as genomic_bench_dataset, nucleotide_transformer_dataset, etc.). Also mention the external data files expected in the “data” directory.
• Configuration & Experiment Management – Illustrate how the project uses Hydra with extensive configuration files (located in “configs/” directories) to control everything from model hyperparameters to training tasks. This is essential to support multiple experiments (e.g., pretraining, finetuning, in-context learning).
• Custom C++/CUDA Operations – Highlight the “csrc” module which implements the performance-critical FFT convolution and related operations. These custom operations integrate with Python and are used by the model layers for efficiency.
• Evaluation & Inference – Represent the “evals” directory that includes scripts for inference (e.g., hg38_inference.py, soft_prompting_genomics.py) and also the “huggingface.py” and “standalone_hyenadna.py” for alternative inference pathways.
• Docker and Submodules – Show that Docker is used to package the entire environment (using Dockerfile) and note that external submodules (like flash-attention) are integrated for additional functionality.
Relationships and Interactions
• The configuration system (Hydra) feeds settings to the training pipeline—influencing model parameters (such as d_model, n_layer, l_max), dataloader settings (batch size, maximum sequence length), and optimizer/hyperparameter settings.
• The training script (train.py) is the central coordinator that invokes the dataloaders (which load data from “data” directories or external sources), instantiates the backbone model (from “src/models/…”), and launches experiments via PyTorch Lightning.
• Custom CUDA extensions and C++ operations (from “csrc”) are built during installation or via Docker and are used by the model layers to provide optimized numerical operations.
• Evaluation scripts and inference modules interact with the trained model (and optionally pretrained weights from HuggingFace) to serve as downstream tasks or generate embeddings/logits.
• Docker images and Dockerfile guarantee a reproducible environment for development and execution, and are used either directly or via containerized experiments.
• There is a clear separation of concerns: configuration management, model definition, data handling, and training orchestration are structured in separate directories.
Architectural Patterns and Design Principles
• The project employs a modular design that decouples configuration, model definition, training routines, and evaluation. Hydra is used to make experiments easily switchable via configuration overlays.
• It uses the “separation of concerns” principle to separate the data pipeline (dataloaders and datasets) from the model logic (“src/models”) and training orchestration (train.py plus task definitions in “src/tasks”).
• Custom performance-critical components (C++/CUDA in “csrc”) are kept distinct so that they can be maintained and updated independently.
• The system is designed to be extendable: new datasets and tasks can be added with minimal changes by following the configuration structure in “configs” and the class registration mechanisms in “src/utils/registry.py”.
Technologies, Frameworks, and Libraries
• PyTorch Lightning for managing training loops and distributed training.
• Hydra for configuration management, with numerous YAML config files for experiments, datasets, models, pipelines, optimizers, etc.
• Docker to encapsulate dependencies and runtime requirements.
• Custom C++/CUDA code (in “csrc”) for FFT convolution and Flash Attention operations.
• External repositories/submodules such as Flash-Attention, and links to HuggingFace pretrained model weights.
• Datasets and evaluation scripts designed to interface with genomics-specific file formats (.fasta, .bed) for genomic data.
Specific Elements
• Configuration Manager (Hydra with configs)
• Training Pipeline / PyTorch Lightning (train.py and tasks)
• Model Architecture (backbone + decoder) with submodules from “src/models/sequence” and “src/models/nn”
• Data Loader & Dataset Modules (in “src/dataloaders”)
• Custom Ops (C++/CUDA in “csrc”) integrated with model layers
• Evaluation Components (scripts in “evals/” and standalone inference modules)
• Docker Environment and Submodule integration
• How configs drive both training and data handling
• Flow of data from raw input files (e.g., fasta) into dataloaders and then into the model
• Model outputs being passed to evaluation modules
• Core training elements (e.g., model and training loop) versus
• Auxiliary components (e.g., Docker setup, custom ops)
• Annotation notes to mention key external references (such as HuggingFace integration, Docker image usage, and submodule updates).