# riedlerm/multimodal_rag_for_industry architecture

> Architecture diagram of the GitHub repository riedlerm/multimodal_rag_for_industry, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/riedlerm/multimodal_rag_for_industry
- Repository: https://github.com/riedlerm/multimodal_rag_for_industry

## Overview

##### Project Type and Purpose
- This is a research‐grade Python toolkit implementing and evaluating Retrieval‐Augmented Generation (RAG) pipelines enhanced with multimodal (text + image) inputs.
- It provides scripts for data extraction (from PDFs), data summarization, various RAG configurations (text‐only, image‐only, multimodal), and evaluation modules using LLMs (GPT‐4V, LLaVA).

##### File Structure Analysis
- Top‐Level:
- img/ – Documentation figures
- sample_data/ – Precomputed data, doc stores, results
- src/ – All source code, subdivided by function
- requirements.txt, README.md, LICENSE
- src/data_extraction: PDF parsing, context reduction
- src/data_summarization: Text/image summary generation
- src/question_answering:
– baseline/
– correct_context_prompting/
– rag/ (text‐only, image‐only, multimodal; separate vs single vector stores)
- src/evaluation: RAG pipeline evaluation scripts, LLM‐based evaluators
- src/utils: Configuration, model loading, logging
- rag_env.py: Centralizes model choices, paths, environment variables

##### README Insights
- Two multimodal RAG architectures:
a) Separate vector stores: CLIP for images vs text embedding model, dual search
b) Combined store: Summarize images into text, single embedding model, unified search
- Usage steps illustrate a linear pipeline: extract → summarize → index/embeddings → RAG → evaluate

a. Major Components
1. Data Ingestion & Preprocessing
- PDF Content Extractor (src/data_extraction/pdf_content_extractor.py)
- Context Reduction (src/data_extraction/context_reduction.py)
2. Summarization
- Text Summarization (optional)
- Image Summaries via multimodal LLM (src/data_summarization/context_summarization.py)
3. Embedding & Vector Stores
- Text Embeddings (e.g., text-embedding-3-small)
- Image Embeddings (CLIP)
- Two vector‐store flavors: separate (Chroma/DocStore for each modality) vs combined
4. Retrieval‐Augmented Generation
- RAG Chains/Pipelines (src/question_answering/rag/…)
- Baseline & Correct Context Prompting as control flows
5. LLM Model Interfaces
- OpenAI (GPT-4V) and LLaVA wrappers (src/utils/model_loading_and_prompting/)
6. Evaluation
- RAG Pipeline Runner (src/evaluation/evaluate_rag_pipeline.py)
- Evaluators (Answer Correctness, Relevancy, Faithfulness)
7. Configuration & Orchestration
- rag_env.py, azure_config.py, logging utils

b. Relationships & Data Flow

c. Architectural Patterns
- Pipeline pattern: sequential ETL‐style stages
- Strategy pattern: multiple RAG strategies (text‐only, image‐only, multimodal)
- Adapter/Wrapper: src/utils/model_loading_and_prompting abstracts different LLMs
- Config‐driven orchestration: rag_env.py centralizes paths & model selections

d. Key Technologies
- Python scripts & modules
- Chroma (or similar) vector store via parquet/bin files
- CLIP embeddings for images
- OpenAI embedding & LLM endpoints (text‐embedding‐3‐small, GPT‐4V)
- LLaVA multimodal model
- LLM‐based evaluation

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion & Preprocessing"
        A1["PDF Content Extractor"]:::process
        A2["Context Reduction"]:::process
    end
    A1 --> A2

    subgraph "Summarization"
        B1["Text Summarization (Optional)"]:::text
        B2["Image Summaries"]:::multi
    end
    A2 --> B1
    A2 --> B2

    subgraph "Embedding & Storage"
        subgraph "Separate Vector Stores"
            C1["Dual RAG Chain"]:::multi
            C2["CLIP Pipeline"]:::image
            C3["Summary Pipeline"]:::text
            C4["Dual Retrieval Logic"]:::multi
        end
        subgraph "Combined Vector Store"
            C5["RAG Chain"]:::multi
            C6["CLIP Pipeline"]:::image
            C7["Summary Pipeline"]:::text
            C8["Retrieval Logic"]:::multi
        end
    end
    B1 --> C1
    B2 --> C2
    B1 --> C3
    B2 --> C3
    B1 --> C5
    B2 --> C6
    B1 --> C7
    B2 --> C7

    subgraph "RAG Pipelines"
        D1["Text-Only RAG Pipeline"]:::text
        D2["Image-Only RAG Pipeline"]:::image
        D3["Multimodal RAG Pipeline"]:::multi
    end
    C4 --> D1
    C4 --> D2
    C4 --> D3
    C8 --> D1
    C8 --> D2
    C8 --> D3

    subgraph "Control Flows"
        subgraph Baseline
            E1["QA Chain"]:::process
            E2["Runner"]:::process
        end
        subgraph Correct Context Prompting
            E3["QA Chain"]:::process
            E4["Text-Only Runner"]:::text
            E5["Image-Only Runner"]:::image
            E6["Multimodal Runner"]:::multi
        end
    end
    D1 --> E2
    D2 --> E2
    D3 --> E2

    subgraph "Model Interfaces / Adapters"
        F1["GPT-4V Wrapper"]:::external
        F2["LLaMA3 Wrapper"]:::external
        F3["LLaVA Wrapper"]:::external
    end
    D1 --> F1
    D2 --> F1
    D3 --> F1
    F2 --> D3
    F3 --> D3

    subgraph "Evaluation"
        G1["Pipeline Runner"]:::evaluation
        G2["Core Evaluation Logic"]:::evaluation
        G3["Evaluator Interface"]:::evaluation
        G4["Base Evaluator"]:::evaluation
        G5["OpenAI Evaluators"]:::external
        G6["LLaVA Evaluators"]:::external
    end
    D1 --> G1
    D2 --> G1
    D3 --> G1
    G1 --> G2
    G2 --> G3
    G3 --> G4
    G4 --> G5
    G4 --> G6

    subgraph "Configuration & Orchestration"
        H1["rag_env.py"]:::config
        H2["Azure Config"]:::config
        H3["Logging Utilities"]:::config
    end
    H1 --> A1
    H1 --> B1
    H1 --> C5
    H1 --> D1
    H1 --> G1

    click A1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/data_extraction/pdf_content_extractor.py"
    click A2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/data_extraction/context_reduction.py"
    click B2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/data_summarization/context_summarization.py"
    click D1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/run_text_only_rag.py"
    click D2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/run_image_only_rag.py"
    click D3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/run_multimodal_rag.py"
    click C1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/separate_vector_stores/dual_rag_chain.py"
    click C2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/separate_vector_stores/dual_rag_pipeline_clip.py"
    click C3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/separate_vector_stores/dual_rag_pipeline_summaries.py"
    click C4 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/separate_vector_stores/dual_retrieval.py"
    click C5 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/single_vector_store/rag_chain.py"
    click C6 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/single_vector_store/rag_pipeline_clip.py"
    click C7 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/single_vector_store/rag_pipeline_summaries.py"
    click C8 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/rag/single_vector_store/retrieval.py"
    click E1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/baseline/qa_chain.py"
    click E2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/baseline/run_baseline.py"
    click E3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/correct_context_prompting/correct_context_qa_chain.py"
    click E4 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/correct_context_prompting/run_text_only_correct_context_qa.py"
    click E5 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/correct_context_prompting/run_image_only_correct_context_qa.py"
    click E6 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/question_answering/correct_context_prompting/run_multimodal_correct_context_qa.py"
    click F1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/utils/model_loading_and_prompting/gpt4.py"
    click F2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/utils/model_loading_and_prompting/llama3.py"
    click F3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/utils/model_loading_and_prompting/llava.py"
    click H1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/rag_env.py"
    click H2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/utils/azure_config.py"
    click H3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/utils/utils_logging.py"
    click G1 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluate_rag_pipeline.py"
    click G2 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluation_module.py"
    click G3 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluators/evaluator_interface.py"
    click G4 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluators/base_evaluator.py"
    click G5 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluators/evaluators_openai.py"
    click G6 "https://github.com/riedlerm/multimodal_rag_for_industry/blob/main/src/evaluation/evaluators/evaluators_llava.py"

    classDef text fill:#cce5ff
    classDef image fill:#d4edda
    classDef multi fill:#f3e6ff
    classDef external stroke-dasharray: 5 5, fill:#fff
    classDef config fill:#ffe6e6
    classDef evaluation fill:#fff3cd
    classDef process fill:#ececec
```
