# cvenhoff/vlm-mapping architecture

> Architecture diagram of the GitHub repository cvenhoff/vlm-mapping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cvenhoff/vlm-mapping
- Repository: https://github.com/cvenhoff/vlm-mapping

## Overview

##### Project type and purpose
- This is a research-oriented machine-learning pipeline for understanding cross-modal feature alignment in vision-language models (VLMs).
- It provides two main workflows:
a. Training a frozen CLIP-Gemma projector via sparse autoencoder (in `train/`).
b. Running a controlled feature-mapping investigation: cache activations, compute reconstruction errors, and evaluate semantic alignment (via `run.sh` and related scripts).

##### File structure
- Top-level scripts:
- `install_data.sh` – downloads or installs datasets
- `run.sh` – orchestrates the full investigation pipeline
- `cache_activations.py` – extracts and caches model activations
- `sae_reconstruction.py` – computes sparsity / reconstruction metrics
- `sae_descriptions_eval.py` – runs semantic alignment experiments via OpenAI
- `utils.py` – shared helper functions
- Training directory (`train/`):
- `train.py` – training loop for projector finetuning via accelerate
- `train.sh` – wrapper to launch `train.py`
- `weights/` – pre-trained model checkpoints (Gemma-2 projector)
- Results directory (`results/`):
- `figures/` – stored PDFs showing evaluation plots
- Configuration:
- `environment.yml` – Conda environment specification
- `.gitignore` – ignored files
- `.env` (not committed) – holds `HF_TOKEN` and `OPENAI_API_KEY`

##### README for additional insights
- Confirms the two workflows (training and investigation)
- Specifies dependencies: PyTorch, Hugging Face CLIP/Gemma loader, OpenAI API
- Describes the high-level steps of the `run.sh` pipeline

##### Components
a. Data & Configuration
– Data installer (`install_data.sh`) → Raw dataset
– Conda environment (`environment.yml`) → Python environment
– Environment variables (`.env`) → HF_TOKEN, OPENAI_API_KEY
b. Training workflow
– Frozen VLM (CLIP + Gemma 2) model
– Sparse Autoencoder training (`train/train.py`)
– Checkpoints stored in `train/weights/`
c. Investigation workflow
– Activation caching (`cache_activations.py`)
– Reconstruction analysis (`sae_reconstruction.py`)
– Semantic alignment evaluation (`sae_descriptions_eval.py`) via HTTP calls to OpenAI
d. Utilities
– Shared helpers (`utils.py`)
e. Results storage
– Metric outputs and plots under `results/`

##### Relationships & interactions
install_data.sh → run.sh → cache_activations → sae_reconstruction → sae_descriptions_eval → results/
- Training branch: install_data.sh → train/train.sh → train.py → weights/
- Both workflows depend on the Conda environment and `.env` variables.
- External dependencies:
- Hugging Face model hub (Gemma 2)
- OpenAI API endpoints

##### Architectural patterns & design principles
- Modular pipeline stages (each Python script does one narrowly scoped task)
- Clear separation between training and analysis
- Use of shell wrappers to orchestrate Python modules
- Externalization of credentials via `.env`

##### Technologies, frameworks & libraries
- Python 3.x with Conda environment
- PyTorch (for model loading, training, reconstruction)
- Hugging Face Accelerate (distributed training orchestration)
- OpenAI Python client (semantic alignment evaluations)
- Shell scripting for workflow orchestration

##### Diagramming recommendations
- Scripts (rectangles)
- Edges with key actions where helpful (e.g., “calls OpenAI API”)

##### Detail
- Script as its own component
- Where shared code (`utils.py`) is imported
- Both on-disk data (checkpoints, cached activations, plots) and in-memory models

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Config & Environment
    subgraph "Config & Environment"
        installData["install_data.sh"]:::script
        envFile["environment.yml"]:::config
        envVars[".env"]:::config
        gitignore[".gitignore"]:::config
        readme["README.md"]:::doc
    end

    %% Training Workflow
    subgraph "Training Workflow"
        trainLaunch["train/train.sh"]:::script
        trainScript["train/train.py"]:::script
        weightsDir["train/weights/"]:::data
        pretrainChk["projector_gemma-2-2b-it_pretrain.pth"]:::data
        finetuneChk["projector_gemma-2-2b-it_finetune.pth"]:::data
    end

    %% Investigation Workflow
    subgraph "Investigation Workflow"
        runScript["run.sh"]:::script
        cacheMod["cache_activations.py"]:::script
        reconModule["sae_reconstruction.py"]:::script
        semEval["sae_descriptions_eval.py"]:::script
        figDir["results/figures/"]:::data
        featurePlot["sae_feature_match_rate_lines.pdf"]:::result
        reconPlot["sae_reconstruction_combined.pdf"]:::result
    end

    %% Utilities and External
    utilsMod["utils.py"]:::util
    extHF["Hugging Face Model Hub"]:::external
    extOpenAI["OpenAI API"]:::external

    %% Data Stores
    rawData["Raw Dataset"]:::data
    cachedActs["Cached Activations"]:::data
    reconMetrics["Reconstruction Metrics"]:::data
    semAlign["Semantic Alignment Results"]:::data

    %% Flows: Training
    installData -->|downloads raw data| rawData
    installData -->|sets up data| trainLaunch
    envFile -.-> trainLaunch
    envVars -.-> trainLaunch
    trainLaunch -->|calls| trainScript
    trainScript -->|saves checkpoints| weightsDir
    weightsDir --> pretrainChk
    weightsDir --> finetuneChk
    extHF -->|provides pretrained model| trainScript

    %% Flows: Investigation
    installData --> runScript
    envFile -.-> runScript
    envVars -.-> runScript
    runScript --> cacheMod
    cacheMod -->|writes| cachedActs
    cacheMod -->|imports| utilsMod
    cacheMod -->|uses pretrained model| extHF
    runScript --> reconModule
    reconModule -->|reads| cachedActs
    reconModule -->|writes| reconMetrics
    reconModule -->|imports| utilsMod
    runScript --> semEval
    semEval -->|reads| reconMetrics
    semEval -->|calls| extOpenAI
    semEval -->|writes| semAlign

    semAlign -->|plots| figDir
    figDir --> featurePlot
    figDir --> reconPlot

    %% Shared Utilities
    trainScript -->|imports| utilsMod
    runScript -->|imports| utilsMod

    %% Click Events
    click installData "https://github.com/cvenhoff/vlm-mapping/blob/main/install_data.sh"
    click envFile "https://github.com/cvenhoff/vlm-mapping/blob/main/environment.yml"
    click gitignore "https://github.com/cvenhoff/vlm-mapping/blob/main/.gitignore"
    click readme "https://github.com/cvenhoff/vlm-mapping/blob/main/README.md"
    click trainLaunch "https://github.com/cvenhoff/vlm-mapping/blob/main/train/train.sh"
    click trainScript "https://github.com/cvenhoff/vlm-mapping/blob/main/train/train.py"
    click weightsDir "https://github.com/cvenhoff/vlm-mapping/tree/main/train/weights/"
    click pretrainChk "https://github.com/cvenhoff/vlm-mapping/blob/main/train/weights/projector_gemma-2-2b-it_pretrain.pth"
    click finetuneChk "https://github.com/cvenhoff/vlm-mapping/blob/main/train/weights/projector_gemma-2-2b-it_finetune.pth"
    click runScript "https://github.com/cvenhoff/vlm-mapping/blob/main/run.sh"
    click cacheMod "https://github.com/cvenhoff/vlm-mapping/blob/main/cache_activations.py"
    click reconModule "https://github.com/cvenhoff/vlm-mapping/blob/main/sae_reconstruction.py"
    click semEval "https://github.com/cvenhoff/vlm-mapping/blob/main/sae_descriptions_eval.py"
    click utilsMod "https://github.com/cvenhoff/vlm-mapping/blob/main/utils.py"
    click figDir "https://github.com/cvenhoff/vlm-mapping/tree/main/results/figures/"
    click featurePlot "https://github.com/cvenhoff/vlm-mapping/blob/main/results/figures/sae_feature_match_rate_lines.pdf"
    click reconPlot "https://github.com/cvenhoff/vlm-mapping/blob/main/results/figures/sae_reconstruction_combined.pdf"

    %% Styles
    classDef script fill:#f9f,stroke:#333,stroke-width:1px
    classDef util fill:#ff9,stroke:#333,stroke-width:1px
    classDef config fill:#9ff,stroke:#333,stroke-width:1px
    classDef data fill:#fcf,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef result fill:#cfc,stroke:#333,stroke-width:1px
    classDef external fill:#ccc,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
    classDef doc fill:#eef,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
