How pku-sec-lab/adapmoe works
Project type and purpose
• This is a full-stack research‐focused inference framework (“AdapMoE”) combining algorithmic innovations (adaptive gating, prefetching, DP‐based cache allocation) with a system implementation targeting efficient Mixture‐of‐Experts (MoE) inference on edge/GPU devices.
• The core purpose: load and execute MoE models with dynamic expert selection and caching, minimize latency, and provide benchmarking pipelines for accuracy/performance evaluation.
File structure
Top level
• run.py – CLI entrypoint for basic inference
• requirements.txt – Python dependencies
• src/ – Core inference engine and supporting modules
• benchmarks/ – Evaluation pipelines (performance & accuracy)
src/ subcomponents
• build_model.py – Model construction (MoE layers, expert count)
• custom_layers.py – Definitions of expert and gating layers
• dp.py – Dynamic programming logic for cache allocation
• expert_cache.py – Cache data structures & allocation manager
• expert_wrapper.py – Wrappers to load, unload experts on demand
• linear_cache.py – Specialized cache for dense components
• linear_wrapper.py – Wrappers for non‐MoE layers
• packing.py – Tensor‐packing utilities for batching
• triton_kernels.py – High‐performance kernels via Triton
• utils.py – Logging, configuration parsing, helper functions
benchmarks/
• run_acc_benchmark.py – Accuracy evaluation harness (lm‐evaluation‐harness)
• chain-of-thought-hub/ – Reproduction scripts & data for chain‐of‐thought MMLU tasks
• various conversion scripts, notebooks, data folders under chain‐of‐thought-hub/MMLU
Insights from the README
• Core innovations: adaptive sensitivity gating, prefetching, DP‐based cache allocation
• Two modes: basic inference (run.py) and accuracy eval (benchmarks)
• External dependencies: PyTorch, Triton, lm‐evaluation‐harness, chain‐of‐thought‐hub
a. Main components
1. CLI Orchestrator (run.py)
2. Model Builder & MoE Layers (build_model.py + custom_layers.py)
3. Gating Manager (adaptive sensitivity logic)
4. DP Cache Optimizer (dp.py)
5. Expert Cache & Wrappers (expert_cache.py + expert_wrapper.py)
6. Triton Kernels & Packing (triton_kernels.py + packing.py)
7. GPU Memory (physical cache space)
8. Benchmark Pipelines (performance vs. accuracy)
9. External Data Stores (MMLU datasets) & Evaluation Libraries
b. Relationships & data flow
• The user invokes run.py → configuration parsed → build_model constructs MoE network
• During inference, each token batch flows through:
Gate Manager → decides which experts to activate
DP Optimizer → allocates GPU cache slots per layer
Expert Cache → loads/prefetches experts into GPU via wrappers
Packed inputs → dispatched to Triton kernels for execution
Outputs collected → next layer / final prediction
• Benchmark modules invoke similar flows but wrap run_mmlu or run_acc_benchmark, reading external data, logging metrics.
c. Architectural patterns & design principles
• Modular: separation of gating, caching, model construction
• Pipeline: clearly staged inference pipeline
• Strategy pattern: pluggable gating strategies (naive vs. sensitivity‐based)
• Cache management: DP‐driven resource allocation
d. Key technologies
• PyTorch for tensor operations
• Triton for custom GPU kernels
• lm‐evaluation‐harness & chain‐of‐thought‐hub for accuracy benchmarks
• Conda/Pip for environment management