# kuleshov-group/bd3lms architecture

> Architecture diagram of the GitHub repository kuleshov-group/bd3lms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kuleshov-group/bd3lms
- Repository: https://github.com/kuleshov-group/bd3lms

## Overview

##### Main Components
- Main Application Entry Point – Represent the role of main.py, which drives training, evaluation, and arbitrary-length sequence generation. This script acts as the central orchestrator.
- Configuration Management – Illustrate the configs/ directory branch. Here, you have subdirectories for algorithm parameters (algo), callbacks, data specifications, noise schedules, learning rate schedulers, model configurations, and strategies. Emphasize how these YAML configuration files are loaded and influence the training and sampling routines.
- Data Loading – Mark dataloader.py as the component responsible for feeding datasets into the training pipeline. Indicate how different datasets (e.g., OpenWebText, LM1B, PTB, etc.) are configured and loaded.
- Diffusion and Noise Scheduling – Include diffusion.py and noise_schedule.py, which orchestrate the forward and reverse diffusion processes and the definition of noise schedules (configurations like loglinear, constant, cosine decay, etc.).
- Models/Network Architectures – Represent the models/ directory that contains various architecture definitions. Note that the implementations support both autoregressive and diffusion techniques (with submodules like autoregressive.py, dit.py for DiT, etc.), and also include a sub-package for HuggingFace-compatible models.
- Utility Functions – Identify utils.py and metrics.py which offer logging, learning rate scheduling, and other helper functions.
- Scripts for Experimentation – Outline the scripts/ directory which includes subdirectories for training, evaluation (ppl evaluation), sample generation (gen_ppl), arbitrary-length generation (var_len), and zero-shot evaluation (zs_ppl). These scripts encapsulate specific experiment pipelines and hook into the main entry point with pre-defined configurations.
- SSD-LM Codebase – Although part of the same repository, the ssd-lm directory functions as a semi-independent component for the SSD-LM baseline, complete with its own scripts, evaluation routines, and logging.

##### Relationships and Interactions
- Flow Control: Start with configuration loading which informs the behavior in main.py. The configurations determine which algorithm (e.g., bd3lm, ar, sedd, mdlm) is run and load parameters for the model, noise schedule, data, and training.
- External Integration: Indicate integration with external platforms and libraries such as HuggingFace (evident in the push_to_hf.py script, the models/hf/ submodule, and provided checkpoint links).

##### Architectural Patterns and Design Principles
- Separation of Concerns: The repository neatly separates configuration, model definitions, training logic, diffusion processes, and experiment scripts.
- Modularity and Extensibility: Components are modular (for instance, switching between model variants or diffusion algorithms via configuration) and allow for easy extension (additional data sources or training routines).
- Pipeline Orchestration: Emphasize that the main.py acts as a controller that integrates various modules, with clear lifecycle stages (data loading → model processing with diffusion → training/evaluation → logging).
- Reproducibility & Experiment Management: The configs/ and scripts/ directories together facilitate reproducible experiments by providing pre-defined settings for diverse evaluations and training modes.

##### Final Assembly and Detailing
- You address the intricate design of supporting various modes through parameterization (e.g., mode=train vs. mode=ppl_eval vs. mode=sample_eval) as indicated by the command-line instructions in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration & Data"
        ConfigManager("Configuration Manager (configs/)")
        DataLoader("Data Loader (dataloader.py)")
    end

    ConfigManager -->|"loadsconfig"| MainController("Main Controller (main.py)")
    DataLoader -->|"providesdata"| MainController

    subgraph "Core Pipeline"
        %% Main Controller is already defined above.
        subgraph "Diffusion Module"
            DiffCore("Diffusion Core (diffusion.py)")
            NoiseSch("Noise Schedule (noise_schedule.py)")
        end
        MainController -->|"invokesdiffusion"| DiffCore
        DiffCore -->|"cooperateswith"| NoiseSch

        subgraph "Modeling"
            ModelArch("Model Architectures (models/)")
            HuggingFace("HuggingFace Integration (models/hf/)")
        end
        MainController -->|"buildsmodel"| ModelArch
        ModelArch -->|"integratesHF"| HuggingFace

        subgraph "Utility Services"
            Utils("Utils (utils.py)")
            Metrics("Metrics (metrics.py)")
        end
        MainController -->|"usesutils"| Utils
        MainController -->|"logsmetrics"| Metrics
    end

    subgraph "Experimentation"
        ExpScripts("Experiment Scripts (scripts/)")
        SSDLM("SSD-LM Module (ssd-lm/)")
    end
    ExpScripts -->|"triggersexperiments"| MainController
    SSDLM -->|"alternativebaseline"| MainController

    click ConfigManager "https://github.com/kuleshov-group/bd3lms/tree/main/configs/"
    click DataLoader "https://github.com/kuleshov-group/bd3lms/blob/main/dataloader.py"
    click MainController "https://github.com/kuleshov-group/bd3lms/blob/main/main.py"
    click DiffCore "https://github.com/kuleshov-group/bd3lms/blob/main/diffusion.py"
    click NoiseSch "https://github.com/kuleshov-group/bd3lms/blob/main/noise_schedule.py"
    click ModelArch "https://github.com/kuleshov-group/bd3lms/tree/main/models/"
    click HuggingFace "https://github.com/kuleshov-group/bd3lms/tree/main/models/hf/"
    click Utils "https://github.com/kuleshov-group/bd3lms/blob/main/utils.py"
    click Metrics "https://github.com/kuleshov-group/bd3lms/blob/main/metrics.py"
    click ExpScripts "https://github.com/kuleshov-group/bd3lms/tree/main/scripts/"
    click SSDLM "https://github.com/kuleshov-group/bd3lms/tree/main/ssd-lm/"

    class ConfigManager,DataLoader config
    class MainController core
    class DiffCore,NoiseSch diffusion
    class ModelArch,HuggingFace model
    class Utils,Metrics utility
    class ExpScripts,SSDLM experiment

    classDef config fill:#C8E6C9,stroke:#43A047,stroke-width:2px;
    classDef core fill:#D1E8FF,stroke:#0277BD,stroke-width:2px;
    classDef diffusion fill:#FFE0B2,stroke:#FB8C00,stroke-width:2px;
    classDef model fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px;
    classDef utility fill:#F8BBD0,stroke:#C2185B,stroke-width:2px;
    classDef experiment fill:#D7CCC8,stroke:#5D4037,stroke-width:2px;
```
