How riedlerm/multimodal_rag_for_industry works
Project Type and Purpose
• This is a research‐grade Python toolkit implementing and evaluating Retrieval‐Augmented Generation (RAG) pipelines enhanced with multimodal (text + image) inputs.
• It provides scripts for data extraction (from PDFs), data summarization, various RAG configurations (text‐only, image‐only, multimodal), and evaluation modules using LLMs (GPT‐4V, LLaVA).
File Structure Analysis
• Top‐Level:
• img/ – Documentation figures
• sample_data/ – Precomputed data, doc stores, results
• src/ – All source code, subdivided by function
• requirements.txt, README.md, LICENSE
• src/data_extraction: PDF parsing, context reduction
• src/data_summarization: Text/image summary generation
• src/question_answering:
– baseline/
– correct_context_prompting/
– rag/ (text‐only, image‐only, multimodal; separate vs single vector stores)
• src/evaluation: RAG pipeline evaluation scripts, LLM‐based evaluators
• src/utils: Configuration, model loading, logging
• rag_env.py: Centralizes model choices, paths, environment variables
README Insights
• Two multimodal RAG architectures:
a) Separate vector stores: CLIP for images vs text embedding model, dual search
b) Combined store: Summarize images into text, single embedding model, unified search
• Usage steps illustrate a linear pipeline: extract → summarize → index/embeddings → RAG → evaluate
a. Major Components
1. Data Ingestion & Preprocessing
• PDF Content Extractor (src/data_extraction/pdf_content_extractor.py)
• Context Reduction (src/data_extraction/context_reduction.py)
2. Summarization
• Text Summarization (optional)
• Image Summaries via multimodal LLM (src/data_summarization/context_summarization.py)
3. Embedding & Vector Stores
• Text Embeddings (e.g., text-embedding-3-small)
• Image Embeddings (CLIP)
• Two vector‐store flavors: separate (Chroma/DocStore for each modality) vs combined
4. Retrieval‐Augmented Generation
• RAG Chains/Pipelines (src/question_answering/rag/…)
• Baseline & Correct Context Prompting as control flows
5. LLM Model Interfaces
• OpenAI (GPT-4V) and LLaVA wrappers (src/utils/model_loading_and_prompting/)
6. Evaluation
• RAG Pipeline Runner (src/evaluation/evaluate_rag_pipeline.py)
• Evaluators (Answer Correctness, Relevancy, Faithfulness)
7. Configuration & Orchestration
• rag_env.py, azure_config.py, logging utils
b. Relationships & Data Flow
c. Architectural Patterns
• Pipeline pattern: sequential ETL‐style stages
• Strategy pattern: multiple RAG strategies (text‐only, image‐only, multimodal)
• Adapter/Wrapper: src/utils/model_loading_and_prompting abstracts different LLMs
• Config‐driven orchestration: rag_env.py centralizes paths & model selections
d. Key Technologies
• Python scripts & modules
• Chroma (or similar) vector store via parquet/bin files
• CLIP embeddings for images
• OpenAI embedding & LLM endpoints (text‐embedding‐3‐small, GPT‐4V)
• LLaVA multimodal model
• LLM‐based evaluation