How vijay-punyamurthy/multimodal-rag-pipeline works
This is a local, multimodal PDF question-answering system: users ingest a document, then retrieve text and relevant diagrams or tables to generate a grounded answer with a local vision-language model. The CLI and Flask UI both invoke the same pipeline. Ingestion separates OCR text, native table extraction, and cropped visuals; querying combines dense and BM25 text recall, reranking, page expansion, and cross-modal image filtering. The graph follows the documented and excerpted calls; external model services are shown separately.
Main entry points: pipeline.py Pipeline · retriever.py Retriever · extractor.py OCR + Tables · config.py Configuration
Components 20
PDF Ingestion
- OCR + Tables
extractor.py - Visual Crops
extractor.py - Text Chunks
extractor.py - Image Chunks
extractor.py - Text Embedder
embeddings.py - CLIP Embedder
embeddings.py
Hybrid Retrieval
- Vector Store
vector_store.py - BM25 Index
bm25_index.py - Retriever
retriever.py - Cross-Encoder Rerank
retriever.py - Image Filter
retriever.py
Answer Generation
- Answer Generator
generator.py - Ollama VLM
- Grounded Answer
generator.py
Other components
- User
- PDF Document
- Configuration
config.py
Connections 27
- User → CLI: runs command
main.py - User → Web UI: uses UI
app.py - PDF Document → CLI: supplies input
main.py - CLI → Pipeline: calls ingest/query
main.py - Web UI → Pipeline: calls ingest/query
app.py - Pipeline → OCR + Tables: extracts PDF
pipeline.py - OCR + Tables → Text Chunks: returns OCR/table text
extractor.py - Visual Crops → Image Chunks: produces image chunks
extractor.py - OCR + Tables → Visual Crops: detects visual content
extractor.py - Pipeline → Text Embedder: embeds text chunks
pipeline.py - Pipeline → CLIP Embedder: embeds images
pipeline.py - Pipeline → Vector Store: adds embeddings
pipeline.py - Pipeline → BM25 Index: builds index
pipeline.py - Pipeline → Retriever: retrieves context
pipeline.py - Retriever → Text Embedder: embeds query
retriever.py - Retriever → CLIP Embedder: embeds image query
retriever.py - Retriever → Vector Store: queries vectors
retriever.py - Retriever → BM25 Index: searches keywords
retriever.py - Retriever → Cross-Encoder Rerank: scores candidates
retriever.py - Retriever → Image Filter: filters image hits
retriever.py - Pipeline → Answer Generator: generates answer
pipeline.py - Answer Generator → Ollama VLM: sends prompt and images
generator.py - Ollama VLM → Grounded Answer: returns response
generator.py - Pipeline → Configuration: uses settings indirectly
- OCR + Tables → Configuration: reads extraction settings
extractor.py - Retriever → Configuration: reads retrieval settings
retriever.py - Answer Generator → Configuration: reads model setting
generator.py