# jieshibo/mole architecture

> Architecture diagram of the GitHub repository jieshibo/mole, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jieshibo/mole
- Repository: https://github.com/jieshibo/mole

## Overview

##### Project type and purpose
- This is a machine-learning research library (MoLE) implementing a novel Mixture-of-Lookup-Experts LLM architecture.
- It provides both pretraining support (under the pretrain folder, built on the Megatron core) and downstream usage via Hugging Face–compatible model files plus a lightweight inference pathway with LUT offloading.

##### File structure
- Top-level:
– modeling_dense.py, modeling_moe.py, modeling_mole.py (training model), modeling_mole_rep.py (inference model)
– reparameterize.py (transforms a trained MoLE checkpoint to an inference-ready LUT representation)
– pretrain/ (contains the Megatron-core engine: data, distributed, model definitions, optimizer, pipeline & tensor parallel, JIT, inference server, training orchestration)
– pretrain/pretrain_gpt.py (entry point for launching a pretraining run)
– pretrain/tokenizer (static JSON files for HF tokenizer config)
– setup.py, pyproject.toml (packaging & dependencies)
- The pretrain/megatron/core subdirectory implements:
– Data pipelines (datasets, tokenizers, indexed datasets, retro retrieval)
– Parallelism layers (tensor, pipeline, distributed checkpointing)
– Transformer building blocks (attention, MLP, custom fusion kernels, MoE routers)
– Model specs for BERT, GPT, T5, Retro, multimodal and vision extensions
– Training loop utilities (arguments, checkpointing, logging, schedulers)
– Inference server & static HTML demo

##### README for additional insights
- The README confirms:
– Purpose: implement, train, reparameterize, and serve MoLE models
– Dependencies: PyTorch 2.0.1, Transformers 4.38.2
– Four Hugging Face model entry points: dense, MoE, MoLE (train), MoLE (inference)
– A dedicated reparameterization step

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Pretraining Flow
    Pretrain["Pretraining Orchestration<br/>(pretrain/pretrain_gpt.py)"]:::pretrain
    click Pretrain "https://github.com/jieshibo/mole/blob/main/pretrain/pretrain_gpt.py"

    %% Megatron Core Engine Subcomponents
    subgraph "Megatron Core Engine" 
        style Megatron Core Engine fill:#e0f7fa,stroke:#006064,stroke-width:2px
        DataLoader["Data Loader & Dataset Registry<br/>(datasets/)"]:::core
        click DataLoader "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/datasets/"
        Parallelism["Parallelism Subsystems"]:::core
        subgraph "Tensor Parallelism" 
            TP["Tensor Parallelism"]:::core
            click TP "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/tensor_parallel/"
        end
        subgraph "Pipeline Parallelism"
            PP["Pipeline Parallelism"]:::core
            click PP "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/pipeline_parallel/"
        end
        subgraph "Distributed Checkpointing"
            DC["Distributed Checkpointing"]:::core
            click DC "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/dist_checkpointing/"
        end
        Transformer["Transformer & MoE Building Blocks"]:::core
        click Transformer "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/transformer/"
        Fusion["Custom Fusion Kernels"]:::core
        click Fusion "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/fusions/"
        Utils["Training Loop Utilities<br/>(utils.py, jit.py)"]:::core
        click Utils "https://github.com/jieshibo/mole/blob/main/pretrain/megatron/core/utils.py"
        click Utils "https://github.com/jieshibo/mole/blob/main/pretrain/megatron/core/jit.py"
        InferenceServer["Inference Server & Demo"]:::core
        click InferenceServer "https://github.com/jieshibo/mole/tree/main/pretrain/megatron/core/inference/"
        click InferenceServer "https://github.com/jieshibo/mole/blob/main/pretrain/megatron/inference/static/index.html"
    end

    %% Checkpoint Storage
    Checkpoint["Checkpoint Storage"]:::storage

    %% Reparameterization
    Reparam["Reparameterization Tool<br/>(reparameterize.py)"]:::conversion
    click Reparam "https://github.com/jieshibo/mole/blob/main/reparameterize.py"

    %% Inference Runtime Subcomponents
    subgraph "Inference Runtime" 
        style Inference Runtime fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
        MoLERep["MoLE Inference Model<br/>(modeling_mole_rep.py)"]:::inference
        click MoLERep "https://github.com/jieshibo/mole/blob/main/modeling_mole_rep.py"
        TokenizerIF["HF Tokenizer Interface<br/>(pretrain/tokenizer/)"]:::inference
        click TokenizerIF "https://github.com/jieshibo/mole/tree/main/pretrain/tokenizer/"
        LUTLogic["Device-map & LUT Offloading Logic"]:::inference
    end

    %% External Dependencies
    subgraph "External Dependencies"
        style External Dependencies fill:#f5f5f5,stroke:#9e9e9e,stroke-width:2px
        PyTorch["PyTorch 2.0"]:::external
        Transformers["HF Transformers"]:::external
        FS["File System (LUT offload)"]:::external
    end

    %% Model Definitions
    subgraph "Model Definitions" 
        style Model Definitions fill:#e8eaf6,stroke:#283593,stroke-width:2px
        Dense["Dense LLM<br/>(modeling_dense.py)"]:::model
        click Dense "https://github.com/jieshibo/mole/blob/main/modeling_dense.py"
        MoE["MoE LLM<br/>(modeling_moe.py)"]:::model
        click MoE "https://github.com/jieshibo/mole/blob/main/modeling_moe.py"
        MoLETrain["MoLE (training)<br/>(modeling_mole.py)"]:::model
        click MoLETrain "https://github.com/jieshibo/mole/blob/main/modeling_mole.py"
    end

    %% Flows
    Pretrain -->|"launches"| DataLoader
    Pretrain -->|"launches"| Parallelism
    Pretrain -->|"launches"| Transformer
    Pretrain -->|"launches"| Fusion
    Pretrain -->|"uses"| Utils
    Pretrain -->|"serves demo"| InferenceServer

    DataLoader -->|"provides data"| Transformer
    Parallelism -->|"coordinates"| Transformer
    Transformer -->|"builds"| Dense
    Transformer -->|"builds"| MoE
    Transformer -->|"builds"| MoLETrain

    Dense -->|"trained by"| Checkpoint
    MoE -->|"trained by"| Checkpoint
    MoLETrain -->|"trained by"| Checkpoint

    Checkpoint -->|"input"| Reparam
    Reparam -->|"outputs inference checkpoint"| MoLERep

    MoLERep -->|"loads"| TokenizerIF
    MoLERep -->|"offloads LUTs to"| FS
    MoLERep -->|"runs on"| PyTorch
    MoLERep -->|"uses"| Transformers

    %% Connections between layers
    MoLERep -->|"generates tokens"| ExternalDependencies

    %% Classes
    classDef pretrain fill:#bbdefb,stroke:#0d47a1,color:#0d47a1
    classDef core fill:#b2ebf2,stroke:#006064,color:#006064
    classDef storage fill:#fff9c4,stroke:#f57f17,color:#f57f17
    classDef conversion fill:#ffe0b2,stroke:#e65100,color:#e65100
    classDef inference fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20
    classDef external fill:#eeeeee,stroke:#616161,color:#616161
    classDef model fill:#e8eaf6,stroke:#283593,color:#283593
```
