How jjovalle99/colpali-rag-app works
Project Type and Purpose
• This is a backend-only Retrieval-Augmented Generation (RAG) service built with FastAPI.
• Purpose: Ingest PDF documents as images, index them in a vector DB (Qdrant) using the ColPali method, and at query time retrieve relevant pages and feed them into a multimodal LLM (Claude Sonnet) for a natural-language response.
High-Level Architecture Components
a. Client / UI
• External web or script that uploads PDFs or issues query requests to the FastAPI endpoints.
b. FastAPI Backend (server.py + src/app/api)
• Endpoints:
• POST /pdf_ingest → kicks off ingestion.
• POST /query → performs similarity search + LLM call.
• Lifespan & state modules hold application startup/shutdown hooks and global singletons (Qdrant client, Supabase client, model loader).
• Dependency layer wires up per-request clients.
c. Ingestion Subsystem
• PDF→JPEG conversion: pdf2image in src/app/colpali/loaders.py
• Image upload: img_uploader service → Supabase object storage
• Embedding: ColQwen 2.5 via colpali-engine → vector embeddings
• Vector write: Qdrant via qdrant_utils in src/app/utils
d. Inference Subsystem
• Query embedding: same ColQwen 2.5 model
• Vector search: Qdrant returns top-K metadata (image keys)
• Image download: img_downloader service from Supabase
• LLM invocation: Claude Sonnet 3.7 via Anthropics
e. Persistence & External Services
• Qdrant (vector database)
• Supabase (object storage for images)
• Anthropics API (LLM calls)
Relationships & Data Flows
[Client] → HTTP POST /pdf_ingest → [FastAPI Ingestion Endpoint] →
→ pdf2image → [Local JPEGs] → img_uploader → [Supabase Storage] →
→ colpali loader → embeddings → qdrant_utils → [Qdrant Collection]
[Client] → HTTP POST /query → [FastAPI Query Endpoint] →
→ embed query → [Qdrant Search] → list of image IDs →
→ img_downloader → [Supabase] → images → Claude Sonnet → [Response JSON]
Architectural Patterns & Principles
• Layered Design: Presentation (API) → Application services → Integration utilities → External systems
• Single Responsibility:
• api/endpoints only parse requests & orchestrate calls
• services handle I/O (upload/download)
• utils handle DB/client wrappers
• colpali/loaders handle conversion & embedding
• Dependency Injection via FastAPI’s Depends for clients (Qdrant, Supabase, Anthropics).
• Containerized Deployment: Dockerfile + Makefile for build/run.