How mlvanguards/evaluate-ocr-rag-systems works
Project Overview and Type
• This project is designed to evaluate OCR and retrieval-augmented generation (RAG) systems.
• Its purpose is to process PDF documents (from the “data” folder), apply OCR and layout analysis (via modules in src/ocr_benchmark and src/pdf_embedding_decider), and subsequently index and retrieve data using a Vespa-powered component (found under src/vespa). There’s also integration with specific schemas/configs (for example, the “colpal” directory and similar test variants).
• The overall structure suggests a pipeline that ingests PDFs, processes them, and then uses search/indexing and possibly inference modules to support RAG.
Main Components and Their Roles
a. Input & Data Layer:
• “data” directory: Contains PDF files to be processed.
• “logs”: For tracking processing events and errors (e.g., logs/failed_documents).
• “colpal” and similar test directories (test, test2, etc.): Provide configuration files like schemas, search-related XML files, and Vespa service definitions.
b. OCR and Preprocessing Layer:
• src/ocr_benchmark: Contains the logic for benchmarking OCR engines. Within this folder:
• “engines” (easy_ocr, config, etc.): Implements the available OCR approaches.
• “utils” (image_processing.py, pdf_processing.py): Handles PDF and image preprocessing.
c. PDF Embedding and Analysis Layer:
• src/pdf_embedding_decider: Focuses on analyzing PDFs, using components such as:
• layout_analyzer.py
• table_detector.py
• text_density_analyzer.py
• visual_element_analyzer.py
• This layer likely decides how to create embeddings or determine key structures for further processing.
d. Vespa Integration Layer:
• src/vespa: Contains “indexing” (with modules like pdf_processor.py, prepare_feed.py, run.py) and “retrieval” (retrieval/run.py) subfolders.
• This module configures and manages Vespa indexing, retrieval, and overall search services.
• Additional modules (vespa_config.yaml, model.py, etc.) suggest that Vespa is a core component in managing the RAG aspect.
e. Additional Modules:
• src/interfaces: Possibly containing abstract definitions or API endpoints (interfaces.py).
• src/prompts: Contains modules like llama_vision.py, suggesting there might be prompt generation for interfacing with LLMs or vision-based inference.
• Main scripts (main.py, vespa_inference.py, vespa_vision_rag.py) act as entry points into various workflows/pipelines.
• Test files (test.py, test_vespa_indexing.py, etc.) ensure that each component is verified independently.
Relationships and Data Flow
• The system starts by ingesting PDF files from the “data” directory.
• The OCR benchmark module extracts text/images using one or more OCR engines.
• Preprocessed PDF data is then passed to the PDF embedding decider, where layout, table, and visual analyses are conducted to determine embedding strategies.
• The processed and annotated content then gets fed into the Vespa indexing module, configuring and storing the data in a search/retrieval-friendly format.
• Retrieval processes (and possibly inference through vespa_inference.py or vespa_vision_rag.py) act on the indexed documents to support RAG operations.
• The “interfaces” and “prompts” modules could be used at integration points to interact with external systems or LLMs.
Architectural Patterns & Design Principles
• Modularization: The project is clearly split into distinct modules (OCR, PDF analysis, Vespa integration, etc.) which interact through well-defined interfaces.
• Pipeline Flow: There’s a sequential processing pipeline (data ingestion → OCR processing → embedding/analysis → indexing / retrieval) that should be clearly delineated.
• Configuration-Driven: The presence of multiple configuration directories and service definitions (e.g., in colpal and test* directories) indicates a reliance on external configuration to drive behavior.