# aniike-t/hybridragllm architecture

> Architecture diagram of the GitHub repository aniike-t/hybridragllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aniike-t/hybridragllm
- Repository: https://github.com/aniike-t/hybridragllm

## Overview

##### Project Type & Purpose
- The project is built for enhanced question answering on large documents. It extracts text from PDFs, processes and chunks the content, builds various indices (traditional inverted index, neural embedding index, and a signature index), and then performs retrieval using a hybrid approach.
- The README instructs on setting up a Gemini key (likely for a neural model) and running the program, further emphasizing the integration of different retrieval paradigms.

##### Main Components
- Data Processing:
- Sub-components:
▸ pdf_extractor (extract text from PDFs)
▸ chunking (segment text into manageable chunks)
- This module is responsible for ingesting and pre-processing raw documents.

- Indexing:
- Sub-modules include:
▸ Inverted Index (traditional IR approach)
▸ Embedding Index (neural retrieval approach)
▸ Signature Index (possible for verification or optimized retrieval)
- These components build different representations of processed data.

- Retrieval:
- Breakdown:
▸ Hybrid Retriever (combines results from various indices)
▸ Query Processor (manages and directs the query flow)
▸ Prompt Builder (likely formats or enriches queries for a neural model)
- This module receives a query, processes it, and retrieves the most relevant information from the indexed data.

- Configuration & Utilities:
▸ config.py contains environment keys and other settings.
▸ Helpers in utils provide supporting functions across the application.

- Orchestration:
- main.py acts as the central point to orchestrate the workflow—from reading inputs, processing data, indexing, and finally invoking the retrieval pipeline.

##### Relationships & Data Flow
- The Prompt Builder plays a role in composing the final query or prompt, possibly integrating context from the retrieval output.
- Finally, the Main Application Flow (main.py) ties these components together, using configuration settings from config.py.

##### Architectural Patterns & Design Principles
- Pipeline Design: The system is clearly partitioned into sequential steps from Data Processing to Indexing and finally Retrieval.
- Modular Design: Each module (data_processing, indexing, retrieval, utils) is separated, making the architecture easier to understand and modify.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Inputs
    PDF["Raw PDF Documents"]:::external
    Query["User Query"]:::external

    %% Data Processing Layer
    subgraph "Data Processing"
        DP1["pdf_extractor"]:::processing
        DP2["Chunking"]:::processing
    end

    %% Indexing Layer
    subgraph "Indexing"
        IDX1["Inverted Index"]:::indexing
        IDX2["Embedding Index"]:::indexing
        IDX3["Signature Index"]:::indexing
    end

    %% Retrieval Layer
    subgraph "Retrieval"
        RET1["Query Processor"]:::retrieval
        RET2["Hybrid Retriever"]:::retrieval
        RET3["Prompt Builder"]:::retrieval
    end

    %% Configuration & Utilities
    subgraph "Configuration & Utilities"
        CFG["Configuration Settings"]:::config
        UT["Helper Functions"]:::utilities
    end

    %% Main Application Flow
    MAIN["Main Application Flow"]:::main

    %% Data Flow Connections
    PDF -->|"ExtractAndChunk"| DP1
    DP1 -->|"ProcessText"| DP2
    DP2 -->|"FeedData"| IDX1
    DP2 -->|"FeedData"| IDX2
    DP2 -->|"FeedData"| IDX3

    Query -->|"RouteQuery"| RET1
    RET1 -->|"DelegateToHybrid"| RET2
    IDX1 -->|"IndexResults"| RET2
    IDX2 -->|"IndexResults"| RET2
    IDX3 -->|"IndexResults"| RET2
    RET2 -->|"BuildFinalPrompt"| RET3
    RET3 -->|"OutputPrompt"| MAIN

    MAIN -->|"InitializeWith"| CFG
    MAIN -->|"Utilize"| UT
    MAIN -->|"StartProcessing"| DP1

    %% Click Events
    click DP1 "https://github.com/aniike-t/hybridragllm/blob/master/data_processing/pdf_extractor.py"
    click DP2 "https://github.com/aniike-t/hybridragllm/blob/master/data_processing/chunking.py"
    click IDX1 "https://github.com/aniike-t/hybridragllm/blob/master/indexing/inverted_index.py"
    click IDX2 "https://github.com/aniike-t/hybridragllm/blob/master/indexing/embedding_index.py"
    click IDX3 "https://github.com/aniike-t/hybridragllm/blob/master/indexing/signature_index.py"
    click RET1 "https://github.com/aniike-t/hybridragllm/blob/master/retrieval/query_processor.py"
    click RET2 "https://github.com/aniike-t/hybridragllm/blob/master/retrieval/hybrid_retriever.py"
    click RET3 "https://github.com/aniike-t/hybridragllm/blob/master/retrieval/prompt_builder.py"
    click CFG "https://github.com/aniike-t/hybridragllm/blob/master/config.py"
    click UT "https://github.com/aniike-t/hybridragllm/blob/master/utils/helpers.py"
    click MAIN "https://github.com/aniike-t/hybridragllm/blob/master/main.py"

    %% Style Definitions
    classDef external fill:#fef3bd,stroke:#e3a21a,stroke-width:2px;
    classDef processing fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef indexing fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef retrieval fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef config fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef utilities fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef main fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
```
