# pku-sec-lab/adapmoe architecture

> Architecture diagram of the GitHub repository pku-sec-lab/adapmoe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pku-sec-lab/adapmoe
- Repository: https://github.com/pku-sec-lab/adapmoe

## Overview

##### Project type and purpose
- This is a full-stack research‐focused inference framework (“AdapMoE”) combining algorithmic innovations (adaptive gating, prefetching, DP‐based cache allocation) with a system implementation targeting efficient Mixture‐of‐Experts (MoE) inference on edge/GPU devices.
- The core purpose: load and execute MoE models with dynamic expert selection and caching, minimize latency, and provide benchmarking pipelines for accuracy/performance evaluation.

##### File structure
Top level
- run.py                – CLI entrypoint for basic inference
- requirements.txt      – Python dependencies
- src/                  – Core inference engine and supporting modules
- benchmarks/           – Evaluation pipelines (performance & accuracy)
src/ subcomponents
- build_model.py        – Model construction (MoE layers, expert count)
- custom_layers.py      – Definitions of expert and gating layers
- dp.py                 – Dynamic programming logic for cache allocation
- expert_cache.py       – Cache data structures & allocation manager
- expert_wrapper.py     – Wrappers to load, unload experts on demand
- linear_cache.py       – Specialized cache for dense components
- linear_wrapper.py     – Wrappers for non‐MoE layers
- packing.py            – Tensor‐packing utilities for batching
- triton_kernels.py     – High‐performance kernels via Triton
- utils.py              – Logging, configuration parsing, helper functions
benchmarks/
- run_acc_benchmark.py – Accuracy evaluation harness (lm‐evaluation‐harness)
- chain-of-thought-hub/ – Reproduction scripts & data for chain‐of‐thought MMLU tasks
- various conversion scripts, notebooks, data folders under chain‐of‐thought-hub/MMLU

##### Insights from the README
- Core innovations: adaptive sensitivity gating, prefetching, DP‐based cache allocation
- Two modes: basic inference (run.py) and accuracy eval (benchmarks)
- External dependencies: PyTorch, Triton, lm‐evaluation‐harness, chain‐of‐thought‐hub

a. Main components
1. CLI Orchestrator (run.py)
2. Model Builder & MoE Layers (build_model.py + custom_layers.py)
3. Gating Manager (adaptive sensitivity logic)
4. DP Cache Optimizer (dp.py)
5. Expert Cache & Wrappers (expert_cache.py + expert_wrapper.py)
6. Triton Kernels & Packing (triton_kernels.py + packing.py)
7. GPU Memory (physical cache space)
8. Benchmark Pipelines (performance vs. accuracy)
9. External Data Stores (MMLU datasets) & Evaluation Libraries

b. Relationships & data flow
- The user invokes run.py → configuration parsed → build_model constructs MoE network
- During inference, each token batch flows through:
Gate Manager → decides which experts to activate
DP Optimizer → allocates GPU cache slots per layer
Expert Cache → loads/prefetches experts into GPU via wrappers
Packed inputs → dispatched to Triton kernels for execution
Outputs collected → next layer / final prediction
- Benchmark modules invoke similar flows but wrap run_mmlu or run_acc_benchmark, reading external data, logging metrics.

c. Architectural patterns & design principles
- Modular: separation of gating, caching, model construction
- Pipeline: clearly staged inference pipeline
- Strategy pattern: pluggable gating strategies (naive vs. sensitivity‐based)
- Cache management: DP‐driven resource allocation

d. Key technologies
- PyTorch for tensor operations
- Triton for custom GPU kernels
- lm‐evaluation‐harness & chain‐of‐thought‐hub for accuracy benchmarks
- Conda/Pip for environment management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Orchestrator
    subgraph "CLI & Orchestrator"
        CLI["CLI Orchestrator"]:::cli
    end
    click CLI "https://github.com/pku-sec-lab/adapmoe/blob/main/run.py"

    %% Model Construction
    subgraph "Model Construction"
        Build["Model Builder"]:::core
        Layers["MoE Layers & Gating Manager"]:::core
        Build --> Layers
    end
    click Build "https://github.com/pku-sec-lab/adapmoe/blob/main/src/build_model.py"
    click Layers "https://github.com/pku-sec-lab/adapmoe/blob/main/src/custom_layers.py"

    %% Runtime Pipeline
    subgraph "Runtime Pipeline"
        Gate["Gating Manager"]:::core
        DP["DP-based Cache Optimizer"]:::caching
        EC["Expert Cache Manager"]:::caching
        EW["Expert Load/Unload Wrappers"]:::caching
        LC["Linear Cache"]:::caching
        LW["Linear Layer Wrappers"]:::caching
        Pack["Tensor Packing Utilities"]:::exec
        TK["High-Performance Triton Kernels"]:::exec
        Gate --> DP --> EC --> EW --> Pack --> TK
    end
    click DP "https://github.com/pku-sec-lab/adapmoe/blob/main/src/dp.py"
    click EC "https://github.com/pku-sec-lab/adapmoe/blob/main/src/expert_cache.py"
    click EW "https://github.com/pku-sec-lab/adapmoe/blob/main/src/expert_wrapper.py"
    click LC "https://github.com/pku-sec-lab/adapmoe/blob/main/src/linear_cache.py"
    click LW "https://github.com/pku-sec-lab/adapmoe/blob/main/src/linear_wrapper.py"
    click Pack "https://github.com/pku-sec-lab/adapmoe/blob/main/src/packing.py"
    click TK "https://github.com/pku-sec-lab/adapmoe/blob/main/src/triton_kernels.py"

    %% GPU Memory
    GPU["GPU Memory (Cache)"]:::storage
    EC -->|load experts| GPU
    EW -->|manage expert load/unload| GPU
    Pack --> TK --> GPU

    %% Results
    Results["Inference Results"]:::output
    GPU --> Results

    %% Common Utilities
    Utils["Common Utilities"]:::utils
    click Utils "https://github.com/pku-sec-lab/adapmoe/blob/main/src/utils.py"
    CLI --> Utils
    Build --> Utils
    Layers --> Utils
    DP --> Utils
    EC --> Utils
    EW --> Utils

    %% Benchmarks
    subgraph "Benchmarks"
        Performance["Perf & Accuracy Benchmark Harness"]:::bench
        COT["Chain-of-Thought / MMLU Evaluation Suite"]:::bench
        Performance --> ResultsB["Benchmark Results"]:::output
        COT --> ResultsB
    end
    click Performance "https://github.com/pku-sec-lab/adapmoe/blob/main/benchmarks/run_acc_benchmark.py"
    click COT "https://github.com/pku-sec-lab/adapmoe/tree/main/benchmarks/chain-of-thought-hub"

    %% External Dependencies
    subgraph "External Dependencies"
        Torch["PyTorch"]:::external
        TritonLib["Triton"]:::external
        EvalLib["lm-evaluation-harness"]:::external
        Datasets["MMLU Datasets"]:::external
    end
    Layers --> Torch
    DP --> Torch
    TK --> TritonLib
    Performance --> EvalLib
    COT --> EvalLib
    COT --> Datasets

    %% Data Flow
    CLI --> Build
    CLI --> Performance
    CLI --> COT

    %% Styles
    classDef cli fill:#B3E5FC,stroke:#0288D1,color:#01579B;
    classDef core fill:#E1F5FE,stroke:#03A9F4,color:#01579B;
    classDef caching fill:#E8F5E9,stroke:#4CAF50,color:#1B5E20;
    classDef exec fill:#F3E5F5,stroke:#9C27B0,color:#4A148C;
    classDef storage fill:#FFF3E0,stroke:#FF9800,color:#E65100;
    classDef utils fill:#FFEBEE,stroke:#E91E63,color:#880E4F;
    classDef bench fill:#FFF9C4,stroke:#FBC02D,color:#F57F17;
    classDef external fill:#ECEFF1,stroke:#607D8B,color:#263238;
    classDef output fill:#FFCDD2,stroke:#F44336,color:#B71C1C;
```
