How marmikchaudhari/moe-interp works
Project type and purpose
• This is an open-source research codebase (“moe-interp”) for analyzing and interpreting Mixture-of-Experts (MoE) language models.
• It ingests large text corpora (math, physics, biology, legal, GitHub), preprocesses them, runs MoE experiments (routing, swapping, zero-out), and generates outputs (model outputs, expert distributions, plots).
File structure
• Top-level directories:
– data-ext: raw and chunked text sources plus preprocessing notebooks and generated .pt files
– archive: historical data, plots, results, and sample notebooks (for reference)
• Key files:
– *.ipynb: experiment definitions (expert-routing.ipynb, swap-experiments.ipynb, zero-out-exp.ipynb, etc.)
– cuda.sh: shell script to launch GPU-accelerated experiments
– .pt files in data-ext/pt: preprocessed expert data for various categories and model variants
– results/ and plots/: generated output (text results, perplexity CSVs, HTML heatmaps)
• No standalone “backend” or “frontend”—instead, a pipeline of data ingestion, preprocessing, model execution, and visualization.
README for insights
• Confirms data sources, experiment types (routing, swapping, zero-out, MLP vs. dense transformer), and output directories.
Detail without overthinking
• Every directory’s role if relevant (data-ext vs. archive)
• Experiment notebooks as separate modules so newcomers can see where to add or modify experiments
• External dependencies (e.g., ArXiv scraper, OLMoE repo) at the boundaries