# openbmb/visrag architecture

> Architecture diagram of the GitHub repository openbmb/visrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/visrag
- Repository: https://github.com/openbmb/visrag

## Overview

##### Project type and purpose
- VisRAG is an open-source, multi-stage machine-learning pipeline for vision-based retrieval-augmented generation (RAG) on documents. It combines a retrieval model (VisRAG-Ret) built on OpenMatch and a generation pipeline (VisRAG-Gen) using off-the-shelf VLMs/LLMs.

##### File structure
- Top-level “src/openmatch”: core library extending OpenMatch for training/evaluating a dense retrieval model over image embeddings.
- “scripts/…”: CLI orchestration for data conversion (`file2img`), model training (`train_retriever`), evaluation (`eval_retriever`), index building & demo pipeline (`demo/visrag_pipeline`), and generation (`generate`).
- “timm_modified”: patched vision‐model library supporting gradient checkpointing for memory‐efficient training.
- “config/deepspeed.json”: distributed‐training settings; “requirements.txt” & “setup.py” define dependencies and packaging.

##### Main components & their roles
A. Data Layer
- Raw multi-page PDFs and other documents
- Scripts to convert files→images (`scripts/file2img`)
- Training/evaluation Q-D pairs in Parquet/JSON under `examples/` or HF repos
B. Preprocessing & Indexing
- OCR pipelines (`scripts/demo/ocr_pipeline/*`) for optional text extraction
- Image embedding & index builder (`scripts/demo/visrag_pipeline/build_index.py`)
C. Retrieval Model (VisRAG-Ret)
- Data collators & datasets: `src/openmatch/dataset/…`
- Model definitions: `src/openmatch/modeling/…` (SigLIP+MiniCPM-V fusion, weighted page selection)
- Training driver (`src/openmatch/driver/train.py`) leveraging Deepspeed and timm_modified
- Evaluation driver (`src/openmatch/driver/eval.py`) producing TREC-format retrieval results
D. Generation Pipeline (VisRAG-Gen)
- Takes top-k retrieved image embeddings → page concatenation or weighted selection → fed into off-the-shelf VLMs (MiniCPM-V, GPT-4o etc.)
- Invocation via `scripts/generate/generate.py` or HuggingFace Space / Colab demo
E. Core Libraries & Dependencies
- `src/openmatch`: modular retriever core
- `timm_modified`: vision modeling support
- Transformers, PyTorch, Deepspeed, FAISS (through OpenMatch), PIL, etc.

##### Relationships & data flow
1. User supplies documents → `file2img` conversion → images
2. Images + optional OCR text → indexed via VisRAG-Ret index builder
3. During querying: query text → tokenizer+model → query embedding
4. Query embedding + FAISS index → top-k image embeddings
5. Retrieved embeddings → generator: concatenated/resampled pages → VLM → final answer
6. CLI scripts glue stages; core library provides reusable modules

##### Architectural patterns & design principles
- Layered architecture: Data → Preprocessing → Indexing → Retrieval → Generation
- Plugin/strategy pattern in modeling: multiple `modeling_*` submodules (SigLIP, MiniCPMV, VisRAG-Ret) can be swapped
- Modular monolith: clear separation between core library (`src/openmatch`), external-modified library (`timm_modified`), and orchestration scripts
- Configuration as code: `config/deepspeed.json`, argparse in `src/openmatch/arguments.py`

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Doc["Raw Documents (PDF, etc.)"]:::data
        File2Img["File2Img Converter"]:::script
        Images["Image Files (.png)"]:::data
    end

    %% Preprocessing & Indexing
    subgraph "Preprocessing & Indexing"
        direction TB
        OCR["OCR Pipeline"]:::script
        IndexBuilder["Index Builder"]:::script
        FAISS["FAISS Index"]:::core
    end

    %% Retrieval Model Training & Evaluation
    subgraph "Retrieval Model (VisRAG-Ret)"
        direction TB
        subgraph "Data Layer Modules"
            direction TB
            DataCollator["data_collator.py"]:::core
            TrainDataset["train_dataset.py"]:::core
            InferenceDataset["inference_dataset.py"]:::core
        end
        subgraph "Model Definitions"
            direction TB
            DenseModel["dense_retrieval_model.py"]:::core
            VisRAGRet["modeling_visrag_ret.py"]:::core
            MiniCPMV["modeling_minicpmv/"]:::core
            SigLIP["modeling_siglip/"]:::core
            WeightedSel["weighted_selection/"]:::core
        end
        TrainDriver["train.py"]:::core
        EvalDriver["eval.py"]:::core
        DeepspeedCfg["deepspeed.json"]:::script
    end

    %% Retrieval Inference
    subgraph "Retrieval Inference"
        direction TB
        Query["Query Text"]:::data
        Tokenizer["Tokenizer + Model"]:::core
        QueryEmb["Query Embedding"]:::core
        VectorSearch["Vector Search (FAISS)"]:::core
        Retrieved["Top-k Embeddings"]:::data
    end

    %% Generation Pipeline
    subgraph "Generation Pipeline (VisRAG-Gen)"
        direction TB
        AnswerDemo["Demo Answer Script"]:::script
        GenerateEntry["generate.py"]:::script
        GenUtils["generation_utils.py"]:::core
        InferenceMod["inference.py"]:::core
        VLM["VLM API (MiniCPM-V/GPT-4o)"]:::external
        FinalAns["Final Answer"]:::data
    end

    %% Core Libraries & Packaging
    subgraph "Core Libraries & Config"
        direction TB
        CoreLib["src/openmatch"]:::core
        TimmMod["timm_modified"]:::core
        SetupPy["setup.py"]:::script
        ReqTxt["requirements.txt"]:::script
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HF["HuggingFace Hub"]:::external
        Colab["Colab Demo"]:::external
        Arxiv["arXiv"]:::external
    end

    %% Connections Data -> Preprocessing
    Doc --> File2Img --> Images
    Images --> OCR
    OCR --> IndexBuilder --> FAISS

    %% Training & Evaluation Flow
    DataCollator --> TrainDriver
    TrainDataset --> TrainDriver
    DeepspeedCfg --> TrainDriver
    TrainDriver --> FAISS
    EvalDriver --> FAISS

    DenseModel --> TrainDriver
    VisRAGRet --> TrainDriver
    MiniCPMV --> TrainDriver
    SigLIP --> TrainDriver
    WeightedSel --> TrainDriver
    CoreLib --> TrainDriver

    %% Retrieval Inference Flow
    Query --> Tokenizer --> QueryEmb --> VectorSearch --> Retrieved

    Retrieved --> AnswerDemo --> GenerateEntry
    GenerateEntry --> GenUtils --> InferenceMod --> VLM --> FinalAns

    %% External Integration
    HF --> GenerateEntry
    Colab --> GenerateEntry
    Arxiv --> File2Img

    %% Packaging Dependencies
    SetupPy --> CoreLib
    ReqTxt --> CoreLib
    TimmMod --> TrainDriver

    %% Click Events
    click File2Img "https://github.com/openbmb/visrag/tree/master/scripts/file2img"
    click OCR "https://github.com/openbmb/visrag/tree/master/scripts/demo/ocr_pipeline"
    click IndexBuilder "https://github.com/openbmb/visrag/blob/master/scripts/demo/visrag_pipeline/build_index.py"
    click AnswerDemo "https://github.com/openbmb/visrag/blob/master/scripts/demo/visrag_pipeline/answer.py"
    click DataCollator "https://github.com/openbmb/visrag/blob/master/src/openmatch/dataset/data_collator.py"
    click TrainDataset "https://github.com/openbmb/visrag/blob/master/src/openmatch/dataset/train_dataset.py"
    click InferenceDataset "https://github.com/openbmb/visrag/blob/master/src/openmatch/dataset/inference_dataset.py"
    click DenseModel "https://github.com/openbmb/visrag/blob/master/src/openmatch/modeling/dense_retrieval_model.py"
    click VisRAGRet "https://github.com/openbmb/visrag/blob/master/src/openmatch/modeling/modeling_visrag_ret/modeling_visrag_ret.py"
    click MiniCPMV "https://github.com/openbmb/visrag/tree/master/src/openmatch/modeling/modeling_minicpmv/"
    click SigLIP "https://github.com/openbmb/visrag/tree/master/src/openmatch/modeling/modeling_siglip/"
    click WeightedSel "https://github.com/openbmb/visrag/tree/master/src/openmatch/modeling/weighted_selection/"
    click TrainDriver "https://github.com/openbmb/visrag/blob/master/src/openmatch/driver/train.py"
    click EvalDriver "https://github.com/openbmb/visrag/blob/master/src/openmatch/driver/eval.py"
    click GenerateEntry "https://github.com/openbmb/visrag/blob/master/scripts/generate/generate.py"
    click GenUtils "https://github.com/openbmb/visrag/blob/master/src/openmatch/generation_utils.py"
    click InferenceMod "https://github.com/openbmb/visrag/blob/master/src/openmatch/inference/inference.py"
    click CoreLib "https://github.com/openbmb/visrag/tree/master/src/openmatch"
    click TimmMod "https://github.com/openbmb/visrag/tree/master/timm_modified"
    click DeepspeedCfg "https://github.com/openbmb/visrag/blob/master/config/deepspeed.json"
    click SetupPy "https://github.com/openbmb/visrag/blob/master/setup.py"
    click ReqTxt "https://github.com/openbmb/visrag/blob/master/requirements.txt"

    %% Styles
    classDef data fill:#ccf,stroke:#339,stroke-width:1px
    classDef core fill:#cfc,stroke:#393,stroke-width:1px
    classDef script fill:#fc9,stroke:#c60,stroke-width:1px
    classDef external fill:#c9c,stroke:#636,stroke-width:1px
```
