How zymrael/savanna works
Core Components
• Config & Launch Infrastructure:
– Show a “Configuration” module encompassing directories like “configs”, “launcher”, “tuner”, and “tools”. These files and scripts (e.g., SLURM jobs, search_config files, launch scripts) are used to specify model parameters, distributed training settings, checkpoint handling, and hyperparameter tuning.
• Data Management & Preprocessing:
– Represent the “Data Layer” that reads and prepares input (from savanna/data, scripts/preprocess_data.py, and other related files). This module handles tokenization (byte–level for DNA/character data) and dataset splits.
• Distributed Training Engine:
– Illustrate the “Training Runtime” provided by the “savanna” package. This is where the distributed training logic lives (e.g., savanna/distributed, savanna/initialize, savanna/mpu for model parallel and data-parallel integration). Indicate integration with external frameworks:
• DeepSpeed – for ZeRO sharding.
• TransformerEngine – for FP8 operations.
• Aspects borrowed from GPT-NeoX (directory structure and configuration).
• Model & Kernel Modules:
– Create a block for “Model Components” (from savanna/model) that includes submodules for different operators (attention, convolutional multi-hybrid operations, hyena variants, positional embeddings, etc.). Also include a submodule for “Kernel Implementations” (e.g., savanna/kernels and embedded Triton source in savanna/kernels/triton_src) which implements custom convolutions and context–parallel algorithms (including “a2a” and “p2p” techniques).
• Tuning & Experimentation:
– Mark out a “Tuner/Hyperparameter Search” component (from the tuner directory) that interacts with the configuration and distributed training modules to adjust model parameters and optimize performance.
• Testing & Regression:
– Represent “Testing” (from tests and launcher/tests) that performs end-to-end regression and unit testing. This ensures correctness of new changes in architecture and distributed aspects.
Relationships and Flow
• Data Flow: Data (raw datasets) is preprocessed (tokenization, sharding) and fed into the training runtime.
• Training Pipeline: The preprocessed data goes into the model (using savanna/model) and custom kernels are called from the kernel modules. Data moves through various distributed ops managed by savanna/distributed and mpu.
• Configuration and Control Flow: The launcher scripts (in launcher/, tools/, and tuner/) read the configuration files, set up the distributed environment (using SLURM scripts and search_configs), and then launch the training job. Transformers such as DeepSpeed and methods from GPT-NeoX are then wired in.
• Custom Kernel Integration: Emphasize that custom Triton kernels are part of the back-end computation flow used in the convolutional operations.
Architectural Patterns & Technologies
• Distributed Systems Patterns: Indicate use of master–worker (or orchestrator–agent) patterns in distributed training (with the launcher coordinating via SLURM and Python scripts).