How prisma-multimodal/vit-prisma works
Project type and purpose
• ViT Prisma is an open-source mechanistic interpretability library for Vision Transformers (ViTs) and CLIP models.
• It provides tools for activation patching, direct logit attribution, attention‐head visualization, training of custom toy ViTs, and evaluation pipelines.
High-level architectural components
a. Core Library (src/vit_prisma/)
b. Data Layer (dataloaders + utils/data_utils)
c. Model Definitions (models + pretrained integrations)
d. Training & Evaluation (training/, model_eval/, sae/)
e. Mechanistic Tools (prisma_tools/)
f. Visualization & Demos (visualization/, demos/)
In addition, show external services:
• Weights & Biases (wandb) for experiment tracking
• Hugging Face (model hub) for checkpoint storage
Relationships and data flows
• Data Layer → Core Library: data loaders feed batched tensors to models.
• Core Library → Model Definitions: builds ViT/CLIP architectures (timm or custom).
• Model → Training Pipeline: trainer, scheduler, early stopping, SAE training.
• Trained Model → Model Evaluation: evaluation scripts produce metrics and activation caches.
• Activation Caches → Mechanistic Tools: logit_lens, hook_point, factored_matrix extract representations.
• Tool outputs → Visualization Layer: Python/JS scripts render attention maps, patch-level lens, interactive HTML.
• Demos (notebooks) sequentially invoke data loaders, models, tools, and visualizations.
Architectural patterns & design principles
• Modular packaging: each logical concern (dataloaders, models, training, tools, visualization) in its own subpackage.
• Separation of concerns:
• configs/ and wandb_configs/ hold hyperparameter and sweep definitions.
• training/ and sae/training/ focus on model fitting and specialized SAE.
• prisma_tools/ implements re-usable hooks and analysis primitives.
• Plugin-style extensibility: new datasets, models, or tools can be dropped into the corresponding folder and auto-discovered by import.
• CI/CD pipeline via GitHub Actions (basic_checks.yml).
Detailed breakdown of major components
• Data Layer
– src/vit_prisma/dataloaders: CIFAR, MNIST, ImageNet, DSprites, induction datasets
– src/vit_prisma/utils/data_utils: low-level file readers, preprocessing pipelines
• Model Layer
– src/vit_prisma/models: base ViT class, timmConfig adapters, OpenCLIP wrappers, toy models
– pretrained loading in models/pretrained.py and prism_tools/loading_from_pretrained.py
• Training & SAE
– src/vit_prisma/training: generic trainer, schedulers, early stopping
– src/vit_prisma/sae/: sparse autoencoder abstractions, SAE training scripts, wandb sweep configs
• Mechanistic Tools
– src/vit_prisma/prisma_tools: hook registration, activation cache, logit lens computation, factored matrix ops
• Visualization
– src/vit_prisma/visualization: Python scripts & JS templates for attention maps, logit lens HTML, patch renderers
– demos/ and docs/ showcase interactive notebooks that combine all pieces
Final recommendations
• Use C4 “containers” for each major folder, and “components” inside for critical classes or scripts.
This structured visual blueprint will give readers an at-a-glance understanding of how ViT Prisma’s modules interact, where the extensibility hooks live, and how external services fit into the overall workflow.