# jacobdunefsky/transcoder_circuits architecture

> Architecture diagram of the GitHub repository jacobdunefsky/transcoder_circuits, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jacobdunefsky/transcoder_circuits
- Repository: https://github.com/jacobdunefsky/transcoder_circuits

## Overview

##### Project Type and Purpose
- This is an open-source research toolkit for reverse-engineering transformer MLP sublayers in large language models using “transcoders.”
- It provides:
- Core libraries for training and applying sparse autoencoders (“SAEs”) and transcoders (sae_training/)
- A set of analysis tools for interpreting circuits (transcoder_circuits/)
- Example scripts and Jupyter notebooks (walkthrough.ipynb, case studies, sweep, interp-comparison)
- Installation and dependency management (setup.sh, requirements.txt)

##### High-Level Components
- Installation Layer
- setup.sh, requirements.txt → installs Python dependencies, downloads pretrained transcoder weights from HuggingFace
- Core Library Layer
- sae_training/ → modules for training SAE/transcoder models, configuration, utilities, a nested geom_median package
- transcoder_circuits/ → modules for analyzing circuits, generating feature dashboards, replacement-context experiments
- Example and Experiment Layer
- train_transcoder.py → script demonstrating transcoder training
- sweep.ipynb, interp-comparison.ipynb → notebooks for large-scale evaluation
- walkthrough.ipynb → guided tutorial of core workflows
- case_study_*.ipynb → deep-dive analyses of specific transcoder features
- Data and Artifacts
- Downloaded weights (HuggingFace)
- Generated activation stores (sae_training/activations_store.py)
- Outputs of notebooks (figures, logs, dashboards)

##### Relationships and Data Flow
1. The user runs setup.sh → installs dependencies and pulls pretrained weights.
2. Core libraries (sae_training) consume model weights → train or load transcoders.
3. Analysis library (transcoder_circuits) takes trained transcoders + model activations → computes interpreter dashboards, feature analyses.
4. Example scripts and notebooks orchestrate calls into the libraries and produce visualizations or evaluation metrics.

##### Architectural Patterns and Principles
- Layered design: installation, core libraries, front-end notebooks/scripts
- Clear separation of concerns: training (sae_training) vs. analysis (transcoder_circuits)
- Extensibility:
- geom_median is a plug-in style subpackage for specialized algorithms
- Users can add new notebooks or scripts to call the libraries

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Installation Layer
    subgraph "Installation Layer"
        Setup["/ setup.sh /"]:::script
        Req["requirements.txt"]:::script
    end

    %% External Resources
    PyPI[(PyPI)]:::external
    HF[(HuggingFace Model Weights)]:::external

    %% Core Library Layer
    subgraph "SAE Library"
        SAE["sae_training/"]:::library
        AS[(Activation Store)]:::data
        subgraph "GeomMedian Plugin"
            GM["geom_median"]:::library
        end
    end

    %% Transcoder Circuits Library
    subgraph "Transcoder Circuits Library"
        TC["transcoder_circuits/"]:::library
    end

    %% Examples & Notebooks
    subgraph "Examples & Notebooks"
        Train["train_transcoder.py"]:::script
        Walk["walkthrough.ipynb"]:::script
        Sweep["sweep.ipynb"]:::script
        IC["interp-comparison.ipynb"]:::script
        CS1["case_study_caught.ipynb"]:::script
        CS2["case_study_citations.ipynb"]:::script
        CS3["case_study_local_context.ipynb"]:::script
        GT["greaterthan.ipynb"]:::script
        RB["restricted blind case studies.ipynb"]:::script
        Figures[(Figures & Dashboards)]:::data
    end

    %% Dependencies and Data Flows
    Setup -->|installs deps| Req
    Req -->|fetches packages| PyPI
    Setup -->|downloads weights| HF
    HF -.->|provides pretrained model| SAE
    SAE -->|stores activations| AS
    SAE -->|optional algorithms| GM
    AS -->|provides activations| TC
    SAE -->|provides transcoders| TC

    %% Example & Notebooks interactions
    Train -->|imports| SAE
    Train -->|imports| TC
    Walk -->|imports| SAE
    Walk -->|imports| TC
    Sweep -->|imports| SAE
    Sweep -->|imports| TC
    IC -->|imports| SAE
    IC -->|imports| TC
    CS1 -->|imports| SAE
    CS1 -->|imports| TC
    CS2 -->|imports| SAE
    CS2 -->|imports| TC
    CS3 -->|imports| SAE
    CS3 -->|imports| TC
    GT -->|imports| SAE
    GT -->|imports| TC
    RB -->|imports| SAE
    RB -->|imports| TC

    %% Outputs
    Train -->|produces| Figures
    Walk -->|produces| Figures
    Sweep -->|produces| Figures
    IC -->|produces| Figures
    CS1 -->|produces| Figures
    CS2 -->|produces| Figures
    CS3 -->|produces| Figures
    GT -->|produces| Figures
    RB -->|produces| Figures

    %% Click Events
    click Setup "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/setup.sh"
    click Req "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/requirements.txt"
    click SAE "https://github.com/jacobdunefsky/transcoder_circuits/tree/master/sae_training/"
    click GM "https://github.com/jacobdunefsky/transcoder_circuits/tree/master/sae_training/geom_median/"
    click TC "https://github.com/jacobdunefsky/transcoder_circuits/tree/master/transcoder_circuits/"
    click Train "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/train_transcoder.py"
    click Walk "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/walkthrough.ipynb"
    click Sweep "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/sweep.ipynb"
    click IC "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/interp-comparison.ipynb"
    click CS1 "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/case_study_caught.ipynb"
    click CS2 "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/case_study_citations.ipynb"
    click CS3 "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/case_study_local_context.ipynb"
    click GT "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/greaterthan.ipynb"
    click RB "https://github.com/jacobdunefsky/transcoder_circuits/blob/master/restricted blind case studies.ipynb"

    %% Styles
    classDef library fill:#cce5ff,stroke:#004085,color:#004085
    classDef script fill:#d4edda,stroke:#155724,color:#155724
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef data fill:#fff3cd,stroke:#856404,color:#856404
```
