How openbmb/visrag works
Project type and purpose
• VisRAG is an open-source, multi-stage machine-learning pipeline for vision-based retrieval-augmented generation (RAG) on documents. It combines a retrieval model (VisRAG-Ret) built on OpenMatch and a generation pipeline (VisRAG-Gen) using off-the-shelf VLMs/LLMs.
File structure
• Top-level “src/openmatch”: core library extending OpenMatch for training/evaluating a dense retrieval model over image embeddings.
• “scripts/…”: CLI orchestration for data conversion (file2img), model training (train_retriever), evaluation (eval_retriever), index building & demo pipeline (demo/visrag_pipeline), and generation (generate).
• “timm_modified”: patched vision‐model library supporting gradient checkpointing for memory‐efficient training.
• “config/deepspeed.json”: distributed‐training settings; “requirements.txt” & “setup.py” define dependencies and packaging.
Main components & their roles
A. Data Layer
• Raw multi-page PDFs and other documents
• Scripts to convert files→images (scripts/file2img)
• Training/evaluation Q-D pairs in Parquet/JSON under examples/ or HF repos
B. Preprocessing & Indexing
• OCR pipelines (scripts/demo/ocr_pipeline/*) for optional text extraction
• Image embedding & index builder (scripts/demo/visrag_pipeline/build_index.py)
C. Retrieval Model (VisRAG-Ret)
• Data collators & datasets: src/openmatch/dataset/…
• Model definitions: src/openmatch/modeling/… (SigLIP+MiniCPM-V fusion, weighted page selection)
• Training driver (src/openmatch/driver/train.py) leveraging Deepspeed and timm_modified
• Evaluation driver (src/openmatch/driver/eval.py) producing TREC-format retrieval results
D. Generation Pipeline (VisRAG-Gen)
• Takes top-k retrieved image embeddings → page concatenation or weighted selection → fed into off-the-shelf VLMs (MiniCPM-V, GPT-4o etc.)
• Invocation via scripts/generate/generate.py or HuggingFace Space / Colab demo
E. Core Libraries & Dependencies
• src/openmatch: modular retriever core
• timm_modified: vision modeling support
• Transformers, PyTorch, Deepspeed, FAISS (through OpenMatch), PIL, etc.
Relationships & data flow
1. User supplies documents → file2img conversion → images
2. Images + optional OCR text → indexed via VisRAG-Ret index builder
3. During querying: query text → tokenizer+model → query embedding
4. Query embedding + FAISS index → top-k image embeddings
5. Retrieved embeddings → generator: concatenated/resampled pages → VLM → final answer
6. CLI scripts glue stages; core library provides reusable modules
Architectural patterns & design principles
• Layered architecture: Data → Preprocessing → Indexing → Retrieval → Generation
• Plugin/strategy pattern in modeling: multiple modeling_* submodules (SigLIP, MiniCPMV, VisRAG-Ret) can be swapped
• Modular monolith: clear separation between core library (src/openmatch), external-modified library (timm_modified), and orchestration scripts
• Configuration as code: config/deepspeed.json, argparse in src/openmatch/arguments.py