# marmikchaudhari/moe-interp architecture

> Architecture diagram of the GitHub repository marmikchaudhari/moe-interp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marmikchaudhari/moe-interp
- Repository: https://github.com/marmikchaudhari/moe-interp

## Overview

##### Project type and purpose
- This is an open-source research codebase (“moe-interp”) for analyzing and interpreting Mixture-of-Experts (MoE) language models.
- It ingests large text corpora (math, physics, biology, legal, GitHub), preprocesses them, runs MoE experiments (routing, swapping, zero-out), and generates outputs (model outputs, expert distributions, plots).

##### File structure
- Top-level directories:
– data-ext: raw and chunked text sources plus preprocessing notebooks and generated .pt files
– archive: historical data, plots, results, and sample notebooks (for reference)
- Key files:
– *.ipynb: experiment definitions (expert-routing.ipynb, swap-experiments.ipynb, zero-out-exp.ipynb, etc.)
– cuda.sh: shell script to launch GPU-accelerated experiments
– .pt files in data-ext/pt: preprocessed expert data for various categories and model variants
– results/ and plots/: generated output (text results, perplexity CSVs, HTML heatmaps)
- No standalone “backend” or “frontend”—instead, a pipeline of data ingestion, preprocessing, model execution, and visualization.

##### README for insights
- Confirms data sources, experiment types (routing, swapping, zero-out, MLP vs. dense transformer), and output directories.

##### Detail without overthinking
- Every directory’s role if relevant (data-ext vs. archive)
- Experiment notebooks as separate modules so newcomers can see where to add or modify experiments
- External dependencies (e.g., ArXiv scraper, OLMoE repo) at the boundaries

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        direction TB
        ArchiveData["Archive Data Sources"]:::dataStore
        ArxivTXT["data-ext/arxiv.txt"]:::dataStore
        GithubTXT["data-ext/github.txt"]:::dataStore
        ArchiveData --> ArxivTXT
        ArchiveData --> GithubTXT
    end

    subgraph "Preprocessing Module"
        direction TB
        Preprocessing["data-ext/data-prep.ipynb"]:::processing
        Chunked["Chunked .txt files"]:::dataStore
        Preprocessing --> Chunked
        ArxivTXT --> Preprocessing
        GithubTXT --> Preprocessing
    end

    Chunked --> PtStorage

    PtStorage["Preprocessed Data Storage (data-ext/pt/)"]:::dataStore

    subgraph "Experiment Runner"
        direction TB
        CudaSH["cuda.sh"]:::compute
        SwapZero["swap-zero-experts.ipynb"]:::processing
        DenseModel["dense-model-exp.ipynb"]:::processing
        CudaSH --> SwapZero
        CudaSH --> DenseModel
        PtStorage --> CudaSH
    end

    CudaSH --> GPU["GPU/CUDA Runtime"]:::compute

    subgraph "Storage & Artifacts"
        direction TB
        PplxCSV["data-ext/pplx-csv/"]:::dataStore
        Results["archive/results/"]:::dataStore
        PlotsArch["archive/plots/"]:::dataStore
        PlotsDeep["archive/plots-deepseek/"]:::dataStore
        CudaSH --> PplxCSV
        CudaSH --> Results
    end

    subgraph "Visualization Module"
        direction TB
        LensVis["moe-lens-visualize.ipynb"]:::visualization
        PlotPplx["plot_pplx.ipynb"]:::visualization
        LensVis --> PlotsArch
        PlotPplx --> PlotsDeep
        PplxCSV --> LensVis
        Results --> PlotPplx
    end

    classDef dataStore fill:#ADD8E6,stroke:#000;
    classDef processing fill:#90EE90,stroke:#000;
    classDef compute fill:#FFA500,stroke:#000;
    classDef visualization fill:#DDA0DD,stroke:#000;

    click ArchiveData "https://github.com/marmikchaudhari/moe-interp/tree/main/archive/data/"
    click ArxivTXT "https://github.com/marmikchaudhari/moe-interp/blob/main/data-ext/arxiv.txt"
    click GithubTXT "https://github.com/marmikchaudhari/moe-interp/blob/main/data-ext/github.txt"
    click Preprocessing "https://github.com/marmikchaudhari/moe-interp/blob/main/data-ext/data-prep.ipynb"
    click Chunked "https://github.com/marmikchaudhari/moe-interp/blob/main/data-ext/arxiv_chunked.txt"
    click PtStorage "https://github.com/marmikchaudhari/moe-interp/tree/main/data-ext/pt/"
    click CudaSH "https://github.com/marmikchaudhari/moe-interp/blob/main/cuda.sh"
    click SwapZero "https://github.com/marmikchaudhari/moe-interp/blob/main/swap-zero-experts.ipynb"
    click DenseModel "https://github.com/marmikchaudhari/moe-interp/blob/main/dense-model-exp.ipynb"
    click GPU "https://github.com/marmikchaudhari/moe-interp/blob/main/cuda.sh"
    click PplxCSV "https://github.com/marmikchaudhari/moe-interp/tree/main/data-ext/pplx-csv/"
    click Results "https://github.com/marmikchaudhari/moe-interp/tree/main/archive/results/"
    click PlotsArch "https://github.com/marmikchaudhari/moe-interp/tree/main/archive/plots/"
    click PlotsDeep "https://github.com/marmikchaudhari/moe-interp/tree/main/archive/plots-deepseek/"
    click LensVis "https://github.com/marmikchaudhari/moe-interp/blob/main/moe-lens-visualize.ipynb"
    click PlotPplx "https://github.com/marmikchaudhari/moe-interp/blob/main/plot_pplx.ipynb"
```
