How cvenhoff/vlm-mapping works
Project type and purpose
• This is a research-oriented machine-learning pipeline for understanding cross-modal feature alignment in vision-language models (VLMs).
• It provides two main workflows:
a. Training a frozen CLIP-Gemma projector via sparse autoencoder (in train/).
b. Running a controlled feature-mapping investigation: cache activations, compute reconstruction errors, and evaluate semantic alignment (via run.sh and related scripts).
File structure
• Top-level scripts:
• install_data.sh – downloads or installs datasets
• run.sh – orchestrates the full investigation pipeline
• cache_activations.py – extracts and caches model activations
• sae_reconstruction.py – computes sparsity / reconstruction metrics
• sae_descriptions_eval.py – runs semantic alignment experiments via OpenAI
• utils.py – shared helper functions
• Training directory (train/):
• train.py – training loop for projector finetuning via accelerate
• train.sh – wrapper to launch train.py
• weights/ – pre-trained model checkpoints (Gemma-2 projector)
• Results directory (results/):
• figures/ – stored PDFs showing evaluation plots
• Configuration:
• environment.yml – Conda environment specification
• .gitignore – ignored files
• .env (not committed) – holds HF_TOKEN and OPENAI_API_KEY
README for additional insights
• Confirms the two workflows (training and investigation)
• Specifies dependencies: PyTorch, Hugging Face CLIP/Gemma loader, OpenAI API
• Describes the high-level steps of the run.sh pipeline
Components
a. Data & Configuration
– Data installer (install_data.sh) → Raw dataset
– Conda environment (environment.yml) → Python environment
– Environment variables (.env) → HF_TOKEN, OPENAI_API_KEY
b. Training workflow
– Frozen VLM (CLIP + Gemma 2) model
– Sparse Autoencoder training (train/train.py)
– Checkpoints stored in train/weights/
c. Investigation workflow
– Activation caching (cache_activations.py)
– Reconstruction analysis (sae_reconstruction.py)
– Semantic alignment evaluation (sae_descriptions_eval.py) via HTTP calls to OpenAI
d. Utilities
– Shared helpers (utils.py)
e. Results storage
– Metric outputs and plots under results/
Relationships & interactions
install_data.sh → run.sh → cache_activations → sae_reconstruction → sae_descriptions_eval → results/
• Training branch: install_data.sh → train/train.sh → train.py → weights/
• Both workflows depend on the Conda environment and .env variables.
• External dependencies:
• Hugging Face model hub (Gemma 2)
• OpenAI API endpoints
Architectural patterns & design principles
• Modular pipeline stages (each Python script does one narrowly scoped task)
• Clear separation between training and analysis
• Use of shell wrappers to orchestrate Python modules
• Externalization of credentials via .env
Technologies, frameworks & libraries
• Python 3.x with Conda environment
• PyTorch (for model loading, training, reconstruction)
• Hugging Face Accelerate (distributed training orchestration)
• OpenAI Python client (semantic alignment evaluations)
• Shell scripting for workflow orchestration
Diagramming recommendations
• Scripts (rectangles)
• Edges with key actions where helpful (e.g., “calls OpenAI API”)
Detail
• Script as its own component
• Where shared code (utils.py) is imported
• Both on-disk data (checkpoints, cached activations, plots) and in-memory models