How jieshibo/mole works
Project type and purpose
• This is a machine-learning research library (MoLE) implementing a novel Mixture-of-Lookup-Experts LLM architecture.
• It provides both pretraining support (under the pretrain folder, built on the Megatron core) and downstream usage via Hugging Face–compatible model files plus a lightweight inference pathway with LUT offloading.
File structure
• Top-level:
– modeling_dense.py, modeling_moe.py, modeling_mole.py (training model), modeling_mole_rep.py (inference model)
– reparameterize.py (transforms a trained MoLE checkpoint to an inference-ready LUT representation)
– pretrain/ (contains the Megatron-core engine: data, distributed, model definitions, optimizer, pipeline & tensor parallel, JIT, inference server, training orchestration)
– pretrain/pretrain_gpt.py (entry point for launching a pretraining run)
– pretrain/tokenizer (static JSON files for HF tokenizer config)
– setup.py, pyproject.toml (packaging & dependencies)
• The pretrain/megatron/core subdirectory implements:
– Data pipelines (datasets, tokenizers, indexed datasets, retro retrieval)
– Parallelism layers (tensor, pipeline, distributed checkpointing)
– Transformer building blocks (attention, MLP, custom fusion kernels, MoE routers)
– Model specs for BERT, GPT, T5, Retro, multimodal and vision extensions
– Training loop utilities (arguments, checkpointing, logging, schedulers)
– Inference server & static HTML demo
README for additional insights
• The README confirms:
– Purpose: implement, train, reparameterize, and serve MoLE models
– Dependencies: PyTorch 2.0.1, Transformers 4.38.2
– Four Hugging Face model entry points: dense, MoE, MoLE (train), MoLE (inference)
– A dedicated reparameterization step