# illuin-tech/colpali architecture

> Architecture diagram of the GitHub repository illuin-tech/colpali, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/illuin-tech/colpali
- Repository: https://github.com/illuin-tech/colpali

## Overview

##### Project Type and Purpose
- This is not a traditional web app but an ML engine focused on multi-modal (text + image) document retrieval.
- Its primary goal is to implement vision language models that compute multi-vector embeddings; it supports training, inference, interpretability (similarity maps), and token pooling.
- The README also indicates that it supports external benchmarking and integration with other services (e.g., Vespa, Weaviate, BentoML) and has a training pipeline via configuration files.

##### Analyzing the File Structure
- The core code lives in the “colpali_engine” directory. Inside, you find well-organized submodules:
- “models”: Contains various model architectures/subvariants (idefics3, idefics_2, paligemma, qwen2, qwen2_5) along with their corresponding processing and modeling files.
- “trainer”: Implements training routines (e.g., colmodel_training.py and contrastive_trainer.py).
- “collators”, “compression”, “interpretability”, “loss”, “utils”: Each represents a key pipeline module—handling data collation, embedding pooling/compression, visual interpretability (similarity maps), loss calculation for training, and general helper functions.
- The “scripts” folder includes example training scripts and configuration files that orchestrate training runs.
- Directories like “tests” show that there is unit/integration testing to cover multiple components.
- CI/CD integration is setup in “.github/workflows” for testing, linting, and publishing – these support continuous integration.

##### Elements
- Clearly labeled components (e.g., Input Processors, Model Components, Loss & Optimization, Token Pooling, Interpretability).
- Integration points such as external benchmarks, CI/CD flow, and optional deployment configurations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing Layer
    subgraph "Data Ingestion & Preprocessing"
        A1["User Input (Images,Text)"]:::input
        A2["Preprocessing Modules"]:::preprocessing
        A3["Processing Utilities"]:::preprocessing
    end

    %% Core ML Engine Layer
    subgraph "Core ML Engine"
        B1["Model Variants"]:::ml
        B2["Embedding Generation"]:::ml
        B3["Token Pooling"]:::pooling
        B4["Interpretability"]:::interpretability
        B5["Loss Computation"]:::loss
    end

    %% Training Pipeline Layer
    subgraph "Training Pipeline"
        C1["Trainer Implementations"]:::training
        C2["Training Scripts/Configs"]:::training
    end

    %% Support & Integration Layer
    subgraph "Support & Integration"
        D1["Utilities"]:::support
        D2["Testing"]:::support
        D3["CI/CD"]:::external
    end

    %% Connections for Data Ingestion & Preprocessing
    A1 --> A2
    A3 -.-> A2

    %% Connections for Core ML Engine
    A2 --> B1
    B1 --> B2
    B2 --> B3
    B2 --> B4
    B2 --> B5

    %% Connections for Training Pipeline
    B5 --> C1
    C1 --> C2
    C2 -->|"update"| B1

    %% Connections for Support & Integration
    D2 --> B1
    D3 --> D2
    D3 --> C1

    %% Click Events
    click A2 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/models/idefics3/colidefics3/processing_colidefics3.py"
    click A3 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/utils/processing_utils.py"
    click B1 "https://github.com/illuin-tech/colpali/tree/main/colpali_engine/models"
    click B3 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/compression/token_pooling.py"
    click B4 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/interpretability/similarity_maps.py"
    click B5 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/loss/bi_encoder_losses.py"
    click C1 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/trainer/colmodel_training.py"
    click C2 "https://github.com/illuin-tech/colpali/blob/main/scripts/train/train_colbert.py"
    click D1 "https://github.com/illuin-tech/colpali/blob/main/colpali_engine/utils/torch_utils.py"
    click D2 "https://github.com/illuin-tech/colpali/tree/main/tests"
    click D3 "https://github.com/illuin-tech/colpali/tree/main/.github/workflows"

    %% Class styling
    classDef input fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef preprocessing fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef ml fill:#C3FDB8,stroke:#333,stroke-width:2px;
    classDef pooling fill:#F7C6A3,stroke:#333,stroke-width:2px;
    classDef interpretability fill:#E2B0FF,stroke:#333,stroke-width:2px;
    classDef loss fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef training fill:#D1CFE2,stroke:#333,stroke-width:2px;
    classDef support fill:#F4E1A6,stroke:#333,stroke-width:2px;
    classDef external fill:#B4C7E7,stroke:#333,stroke-width:2px;
```
