# pat-jj/deepretrieval architecture

> Architecture diagram of the GitHub repository pat-jj/deepretrieval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pat-jj/deepretrieval
- Repository: https://github.com/pat-jj/deepretrieval

## Overview

##### Main Components and Their Interactions
- Data Ingestion/Preprocessing:
– Represent modules under “code/data_preprocess” which contain scripts for processing various datasets (PubMed, clinical trials, BM25 and Dense configurations, SQL-based data, etc.).
– Include two paths: (a) direct data download from Huggingface (preprocessed datasets) and (b) local raw data processing option.
- Core Training Pipeline and Reinforcement Learning:
– Show the RL training components under “code/verl” and “code/src/eval”, which harness distributed training (with scripts in “code/scripts/train”) and evaluation (in “code/scripts/eval”).
– Highlight that the LLM generates reasoning steps (chain-of-thought) for query generation, and it receives rewards based on retrieval performance. Indicate the reward function modules under “code/verl/utils/reward_score” where different retrieval metrics are computed.
- Model Serving and Evaluation:
– Include evaluation components under “code/examples”, “code/src/eval”, and search engine integration modules under “code/src/eval/SearchEngine”. The system interacts with external APIs (e.g., PubMed API) to benchmark query performance.
- Distributed and Parallel Components:
– Illustrate the distributed training elements, including Ray-based orchestration (directories like “code/tests/ray” and “code/verl/single_controller/ray”) and Megatron distributed practices (see “code/verl/models/llama/megatron” and associated worker/actor structures).
- External Dependencies and Infrastructure:
– Note the use of external libraries (PyTorch, vllm, Ray, pyserini, FAISS, Java via install-jdk, etc.) that support computation, retrieval, and training setups.
– Include Docker containerization (located in “code/docker”) which is used for deployment and ensuring environment consistency.

##### Key Architectural Patterns and Design Principles
- Modular Design:
– The project separates data preprocessing, training, evaluation, and model-serving into distinct modules, making it easier to maintain and extend.
- Distributed and Parallel Architecture:
– Emphasize the use of distributed training strategies using Ray and Megatron. This is visible in both the “verl” (model training and orchestration) and “tests” directories, which contain worker and orchestration scripts.
- Extensibility and Testability:
– With dedicated “tests” directories for e2e, sanity, rollout, and GPU utilities, the architecture supports comprehensive testing. Also, the “docs” and “examples” folders provide clear usage examples and configuration variants.
- Data-Driven Reinforcement Learning:
– The training loop centers on LLM query generation with inbuilt chain-of-thought reasoning, using retrieval metrics as training rewards. Illustrate this feedback loop between the model’s output and the reward computation module.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion & Preprocessing"
        A1("Huggingface Direct Download"):::data
        A2("Local Raw Processing"):::data
        A3("Preprocessed Data"):::data
        A1 --> A3
        A2 --> A3
    end

    subgraph "Reinforcement Learning Trainer"
        B1("LLM Query Generation (Chain-of-Thought)"):::training
        B2("RL Training Logic"):::training
        B3("Reward Computation"):::training
        B1 --> B2
        B2 --> B3
    end

    subgraph "Evaluation & Search Engine Integration"
        C1("Evaluation Scripts"):::eval
        C2("Search Engine API Integration"):::eval
    end

    subgraph "Distributed Training & Orchestration"
        D1("Ray-based Orchestration"):::dist
        D2("Megatron Distributed Training"):::dist
        D3("Ray Controller"):::dist
    end

    subgraph "Infrastructure & Deployment"
        E1("Docker Containerization"):::infra
    end

    %% Data Flow Connections
    A3 -->|"produces_data"| B1
    B1 -->|"sends_query"| C1
    B1 -->|"sends_query"| C2
    C1 -->|"feedback"| B2
    C2 -->|"feedback"| B2
    B2 -->|"orchestrates"| D3
    B2 -->|"orchestrates"| D1
    B2 -->|"orchestrates"| D2
    D1 -->|"deployed_in"| E1
    D2 -->|"deployed_in"| E1
    D3 -->|"deployed_in"| E1

    %% Click Events
    click A3 "https://github.com/pat-jj/deepretrieval/tree/main/code/data_preprocess"
    click B1 "https://github.com/pat-jj/deepretrieval/tree/main/code/scripts/train"
    click B2 "https://github.com/pat-jj/deepretrieval/tree/main/code/verl"
    click B3 "https://github.com/pat-jj/deepretrieval/tree/main/code/verl/utils/reward_score"
    click C1 "https://github.com/pat-jj/deepretrieval/tree/main/code/src/eval"
    click C2 "https://github.com/pat-jj/deepretrieval/tree/main/code/src/eval/SearchEngine"
    click D1 "https://github.com/pat-jj/deepretrieval/tree/main/code/tests/ray"
    click D2 "https://github.com/pat-jj/deepretrieval/tree/main/code/verl/models/llama/megatron"
    click D3 "https://github.com/pat-jj/deepretrieval/tree/main/code/verl/single_controller/ray"
    click E1 "https://github.com/pat-jj/deepretrieval/tree/main/code/docker"

    %% Styles
    classDef data fill:#f7d794,stroke:#000,stroke-width:2px;
    classDef training fill:#bad1f2,stroke:#000,stroke-width:2px;
    classDef eval fill:#f2b8d5,stroke:#000,stroke-width:2px;
    classDef dist fill:#d4f2b8,stroke:#000,stroke-width:2px;
    classDef infra fill:#f2e9b8,stroke:#000,stroke-width:2px;
```
