# prisma-multimodal/vit-prisma architecture

> Architecture diagram of the GitHub repository prisma-multimodal/vit-prisma, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prisma-multimodal/vit-prisma
- Repository: https://github.com/prisma-multimodal/vit-prisma

## Overview

##### Project type and purpose
- ViT Prisma is an open-source mechanistic interpretability library for Vision Transformers (ViTs) and CLIP models.
- It provides tools for activation patching, direct logit attribution, attention‐head visualization, training of custom toy ViTs, and evaluation pipelines.

##### High-level architectural components
a. Core Library (src/vit_prisma/)
b. Data Layer (dataloaders + utils/data_utils)
c. Model Definitions (models + pretrained integrations)
d. Training & Evaluation (training/, model_eval/, sae/)
e. Mechanistic Tools (prisma_tools/)
f. Visualization & Demos (visualization/, demos/)
In addition, show external services:
- Weights & Biases (wandb) for experiment tracking
- Hugging Face (model hub) for checkpoint storage

##### Relationships and data flows
- Data Layer → Core Library: data loaders feed batched tensors to models.
- Core Library → Model Definitions: builds ViT/CLIP architectures (timm or custom).
- Model → Training Pipeline: trainer, scheduler, early stopping, SAE training.
- Trained Model → Model Evaluation: evaluation scripts produce metrics and activation caches.
- Activation Caches → Mechanistic Tools: logit_lens, hook_point, factored_matrix extract representations.
- Tool outputs → Visualization Layer: Python/JS scripts render attention maps, patch-level lens, interactive HTML.
- Demos (notebooks) sequentially invoke data loaders, models, tools, and visualizations.

##### Architectural patterns & design principles
- Modular packaging: each logical concern (dataloaders, models, training, tools, visualization) in its own subpackage.
- Separation of concerns:
- configs/ and wandb_configs/ hold hyperparameter and sweep definitions.
- training/ and sae/training/ focus on model fitting and specialized SAE.
- prisma_tools/ implements re-usable hooks and analysis primitives.
- Plugin-style extensibility: new datasets, models, or tools can be dropped into the corresponding folder and auto-discovered by import.
- CI/CD pipeline via GitHub Actions (basic_checks.yml).

##### Detailed breakdown of major components
- Data Layer
– src/vit_prisma/dataloaders: CIFAR, MNIST, ImageNet, DSprites, induction datasets
– src/vit_prisma/utils/data_utils: low-level file readers, preprocessing pipelines
- Model Layer
– src/vit_prisma/models: base ViT class, timmConfig adapters, OpenCLIP wrappers, toy models
– pretrained loading in models/pretrained.py and prism_tools/loading_from_pretrained.py
- Training & SAE
– src/vit_prisma/training: generic trainer, schedulers, early stopping
– src/vit_prisma/sae/: sparse autoencoder abstractions, SAE training scripts, wandb sweep configs
- Mechanistic Tools
– src/vit_prisma/prisma_tools: hook registration, activation cache, logit lens computation, factored matrix ops
- Visualization
– src/vit_prisma/visualization: Python scripts & JS templates for attention maps, logit lens HTML, patch renderers
– demos/ and docs/ showcase interactive notebooks that combine all pieces

##### Final recommendations
- Use C4 “containers” for each major folder, and “components” inside for critical classes or scripts.

This structured visual blueprint will give readers an at-a-glance understanding of how ViT Prisma’s modules interact, where the extensibility hooks live, and how external services fit into the overall workflow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "ViT Prisma Package"
        direction TB

        subgraph "Core Library" 
            direction TB
            Core["src/vit_prisma/"]:::core
        end

        subgraph "Data Layer"
            direction TB
            Dload["src/vit_prisma/dataloaders/"]:::data
            Dutil["src/vit_prisma/utils/data_utils/"]:::data
        end

        subgraph "Model Definitions"
            direction TB
            ModelCore["src/vit_prisma/models/"]:::model
            Pretrained["pretrained.py"]:::model
            OpenCLIP["open_clip_models.py"]:::model
        end

        subgraph "Training & Evaluation"
            direction TB
            Train["src/vit_prisma/training/"]:::train
            Eval["src/vit_prisma/model_eval/"]:::train
            subgraph "SAE"
                direction TB
                SAECore["src/vit_prisma/sae/"]:::train
                SAETrain["src/vit_prisma/sae/training/"]:::train
            end
        end

        subgraph "Mechanistic Tools"
            direction TB
            Tools["src/vit_prisma/prisma_tools/"]:::tools
        end

        subgraph "Visualization & Demos"
            direction TB
            Viz["src/vit_prisma/visualization/"]:::viz
            Demos["demos/"]:::viz
        end
    end

    Wandb["Weights & Biases"]:::ext
    HF["Hugging Face Model Hub"]:::ext

    %% Relationships
    Dload -->|"images→tensors"| Core
    Dutil -->|"preprocessed→tensors"| Core
    Core -->|"builds architectures"| ModelCore
    ModelCore -->|loads| Pretrained
    ModelCore -->|wraps| OpenCLIP
    ModelCore -->|"model→trainer"| Train
    Train -->|"trained model"| Eval
    Eval -->|"activations→cache"| Tools
    Tools -->|"tool outputs"| Viz
    Demos -.->|"invoke data loaders"| Dload
    Demos -.->|"invoke models"| ModelCore
    Demos -.->|"invoke tools"| Tools
    Demos -.->|"render viz"| Viz
    Train -->|metrics→wandb| Wandb
    SAETrain -->|sweep configs→wandb| Wandb
    Pretrained -.->|checkpoint storage| HF
    OpenCLIP -.->|checkpoint storage| HF

    %% Click Events
    click Core "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/"
    click Dload "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/dataloaders/"
    click Dutil "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/utils/data_utils/"
    click ModelCore "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/models/"
    click Pretrained "https://github.com/prisma-multimodal/vit-prisma/blob/main/src/vit_prisma/models/pretrained.py"
    click OpenCLIP "https://github.com/prisma-multimodal/vit-prisma/blob/main/src/vit_prisma/models/open_clip_models.py"
    click Train "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/training/"
    click Eval "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/model_eval/"
    click SAECore "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/sae/"
    click SAETrain "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/sae/training/"
    click Tools "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/prisma_tools/"
    click Viz "https://github.com/prisma-multimodal/vit-prisma/tree/main/src/vit_prisma/visualization/"
    click Demos "https://github.com/prisma-multimodal/vit-prisma/tree/main/demos/"

    %% Styles
    classDef core fill:#88CCFF,stroke:#000,stroke-width:1px;
    classDef data fill:#AAFF88,stroke:#000,stroke-width:1px;
    classDef model fill:#FFCC88,stroke:#000,stroke-width:1px;
    classDef train fill:#FFAA00,stroke:#000,stroke-width:1px;
    classDef tools fill:#CC88FF,stroke:#000,stroke-width:1px;
    classDef viz fill:#FFFF88,stroke:#000,stroke-width:1px;
    classDef ext fill:#DDDDDD,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5;
```
