# hunter-ddm/stablemoe architecture

> Architecture diagram of the GitHub repository hunter-ddm/stablemoe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hunter-ddm/stablemoe
- Repository: https://github.com/hunter-ddm/stablemoe

## Overview

##### Project Type and Purpose
- This is an open-source extension library built on PyTorch Fairseq to implement “StableMoE,” a stable routing strategy for Mixture-of-Experts.
- It combines library code (the customized Fairseq package), command-line interfaces (`train.py`, `fairseq_cli`), example scripts, and tests to enable training and evaluation of MoE language models at scale.

##### File Structure Analysis
- Top-level:
- setup.py, pyproject.toml, build/ → package installation & compiled Cython/C++ extensions
- docs/ → user/developer documentation (Sphinx)
- examples/ → ready-to-run MoE and related tasks
- fairseq/ → core library code, organized by subpackages (data, models, modules, tasks, distributed, optim, criterions, etc.)
- fairseq_cli/ & train.py → command-line front ends
- scripts/ → utility scripts (data prep, scoring, BLEU)
- tests/ → unit and integration tests
- The `build/` directory contains compiled binaries and Cython wrappers under `fairseq/clib` for optimized routines (BLEU, assignment, CUDA kernels).

##### Key Insights from README
- Installation extends base Fairseq via `pip install -e .` and `build_ext --inplace`.
- Users preprocess data with `fairseq-preprocess`.
- Training is done via `train.py` leveraging `torch.distributed.launch` across GPUs.
- Configuration through Fairseq’s YAML registry (`config/model`, `config/optim`) and CLI flags (`--moe-type`, `--assignment-algorithm`).

##### Diagramming the Architecture
a. Main Components
- Command-Line Layer:
– `train.py` / `fairseq_cli` entry points
– Argument parser, configuration loader (Hydra / YAML)
- Core Library (customized Fairseq):
– Data Pipeline: `fairseq/data/*`, `preprocess` commands
– Tasks: `fairseq/tasks/language_modeling.py` with MoE hooks
– Models & Modules:
· Standard Transformers (`fairseq/models/transformer*`)
· MoE Layers & Routing (`fairseq/modules/moe_layer.py`, customized assignment in `fairseq/clib`)
– Criterions & Loss: `xentropy_aux`, `balance-loss`
– Optimizers & Schedulers: `fairseq/optim/*`, polynomial_decay, composite losses
– Distributed Training: `fairseq/distributed/*`, FSDP, DDP, torch.distributed
– Registry / Plugin Mechanism: dynamic loading of archs and tasks
- Build Extensions: C++/CUDA kernels in `fairseq/clib` for fast assignment (GA) and BLEU scoring
- External Services & Data: Storage for shards of preprocessed data (LM shards), checkpoint directory, tensorboard
- Examples & Scripts: usage patterns, data prep, evaluation shells
- Tests: ensure correctness across GPU/CPU, distributed backends

b. Relationships and Interactions
1. User invokes `python train.py` with CLI flags →
2. CLI parses flags, locates YAML configs, registers architecture via Fairseq registry →
3. Data Pipeline reads shards via `fairseq-preprocess` outputs →
4. Task object (`LanguageModelingTask`) sets up Datasets and Batching →
5. Model instantiation: Transformer + MoE layers →
6. Forward pass: modules (MoE router + experts) with routing decisions (C++/CUDA) →
7. Loss computed by criterions (`xentropy_aux` + balance) →
8. Optimizer step (Adam, FSDP) with scheduler →
9. Distributed synchronization via DDP or FSDP →
10. Checkpointing & TensorBoard logging →
11. Evaluation invoked via `fairseq_cli/eval_lm.py`

c. Architectural Patterns & Principles
- Modular plugin architecture: new MoE layers register with Fairseq’s module and model registry
- Separation of concerns: data → task → model → criterion → optimizer → distributed
- High performance through Cython/C++ extensions and CUDA kernels for critical loops
- Config-driven design: YAML files in `config/` drive reproducibility and hyperparameter management

d. Key Technologies & Frameworks
- Python 3.6+ with editable pip install
- PyTorch & torch.distributed (DDP, FSDP)
- Fairseq core library (customized)
- Cython & C++/CUDA for performance (`fairseq/clib`)
- Hydra or custom YAML config loader
- Sphinx for docs, pytest for tests

## Diagram (Mermaid)

```mermaid
graph TD
    %% CLI Layer
    subgraph "Command-Line Layer" 
        CLI1["train.py"]:::cli
        CLI2["fairseq_cli/train.py"]:::cli
        CLI3["fairseq_cli/preprocess.py"]:::cli
        CLI4["fairseq_cli/eval_lm.py"]:::cli
    end

    %% Configuration Registry
    CFG["Configuration Registry<br/>(config/, YAML)"]:::python

    %% Core Library
    subgraph "Core Library (fairseq/)" 
        REG["Registry & Plugin Mechanism<br/>fairseq/registry.py"]:::python
        subgraph "Data Pipeline" 
            DATA["fairseq/data/"]:::python
        end
        subgraph "Tasks"
            TASK["LanguageModelingTask<br/>fairseq/tasks/language_modeling.py"]:::python
        end
        subgraph "Models & Modules"
            TRANS1["Transformer LM<br/>fairseq/models/transformer_lm.py"]:::python
            TRANS2["Transformer<br/>fairseq/models/transformer.py"]:::python
            MOE["MoE Layer & Routing<br/>fairseq/modules/moe_layer.py"]:::python
        end
        subgraph "Criterions"
            CRIT1["XEntropy with Aux Loss<br/>fairseq/criterions/xentropy_with_aux_loss.py"]:::python
            CRIT2["Composite Loss<br/>fairseq/criterions/composite_loss.py"]:::python
        end
        subgraph "Optimizers & Schedulers"
            OPT1["Adam<br/>fairseq/optim/adam.py"]:::python
            OPT2["Composite Optimizer<br/>fairseq/optim/composite.py"]:::python
            SCH["Polynomial Decay<br/>fairseq/optim/lr_scheduler/polynomial_decay_schedule.py"]:::python
        end
        subgraph "Distributed Training"
            DIST1["Legacy DDP<br/>fairseq/distributed/legacy_distributed_data_parallel.py"]:::python
            DIST2["FSDP<br/>fairseq/distributed/fully_sharded_data_parallel.py"]:::python
        end
    end

    %% Build Extensions
    subgraph "Build Extensions (Cython/C++/CUDA)" 
        EXT1["balanced_assignment.cpp<br/>fairseq/clib/libbase/balanced_assignment.cpp"]:::ext
        EXT2["libbleu.cpp<br/>fairseq/clib/libbleu/libbleu.cpp"]:::ext
        EXT3["edit_dist.cpp<br/>fairseq/clib/libnat/edit_dist.cpp"]:::ext
        EXT4["ngram_repeat_block_cuda.cpp/.cu<br/>fairseq/clib/cuda/ngram_repeat_block_cuda.cpp & .cu"]:::ext
    end

    %% Peripheral Components
    EXAMPLES["Examples (MoE Usage)<br/>examples/translation_moe/..."]:::peripheral
    SCRIPTS["Utility Scripts<br/>scripts/"]:::peripheral
    TESTS["Tests<br/>tests/"]:::peripheral
    DOCS["Documentation<br/>docs/"]:::peripheral
    STORAGE["External Storage & Logs"]:::peripheral

    %% Data Flow
    CLI1 -->|"loads config"| CFG
    CLI2 --> CFG
    CLI3 --> DATA
    CLI4 --> TASK

    CFG --> REG
    REG --> TASK
    TASK --> DATA
    TASK --> TRANS1
    TASK --> TRANS2
    TRANS1 --> MOE
    TRANS2 --> MOE

    MOE -->|calls C++/CUDA| EXT1
    MOE --> EXT4

    TASK --> CRIT1
    TASK --> CRIT2
    CRIT1 --> OPT1
    CRIT2 --> OPT2
    OPT1 --> SCH
    OPT2 --> SCH

    SCH --> DIST1
    SCH --> DIST2
    DIST1 --> STORAGE
    DIST2 --> STORAGE

    TASK --> STORAGE
    CLI1 --> STORAGE

    %% Peripheral usage
    EXAMPLES --> TASK
    SCRIPTS --> DATA
    TESTS --> TASK
    DOCS --> REG

    %% Click Events
    click CLI1 "https://github.com/hunter-ddm/stablemoe/blob/main/train.py"
    click CLI2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq_cli/train.py"
    click CLI3 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq_cli/preprocess.py"
    click CLI4 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq_cli/eval_lm.py"
    click CFG "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/config/config.yaml"
    click DATA "https://github.com/hunter-ddm/stablemoe/tree/main/fairseq/data/"
    click TASK "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/tasks/language_modeling.py"
    click TRANS1 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/models/transformer_lm.py"
    click TRANS2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/models/transformer.py"
    click MOE "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/modules/moe_layer.py"
    click EXT1 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/clib/libbase/balanced_assignment.cpp"
    click EXT2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/clib/libbleu/libbleu.cpp"
    click EXT3 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/clib/libnat/edit_dist.cpp"
    click EXT4 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/clib/cuda/ngram_repeat_block_cuda.cpp"
    click CRIT1 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/criterions/xentropy_with_aux_loss.py"
    click CRIT2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/criterions/composite_loss.py"
    click OPT1 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/optim/adam.py"
    click OPT2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/optim/composite.py"
    click SCH "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/optim/lr_scheduler/polynomial_decay_schedule.py"
    click DIST1 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/distributed/legacy_distributed_data_parallel.py"
    click DIST2 "https://github.com/hunter-ddm/stablemoe/blob/main/fairseq/distributed/fully_sharded_data_parallel.py"
    click EXAMPLES "https://github.com/hunter-ddm/stablemoe/blob/main/examples/translation_moe/translation_moe.py"
    click SCRIPTS "https://github.com/hunter-ddm/stablemoe/tree/main/scripts/"
    click TESTS "https://github.com/hunter-ddm/stablemoe/tree/main/tests/"
    click DOCS "https://github.com/hunter-ddm/stablemoe/tree/main/docs/"

    %% Styles
    classDef python fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef cli fill:#C8F7C5,stroke:#333,stroke-width:1px
    classDef ext fill:#FFD8A8,stroke:#333,stroke-width:1px
    classDef peripheral fill:#E3D5FF,stroke:#333,stroke-width:1px
```
