# levuloihust99/sr-dpr architecture

> Architecture diagram of the GitHub repository levuloihust99/sr-dpr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/levuloihust99/sr-dpr
- Repository: https://github.com/levuloihust99/sr-dpr

## Overview

##### Main Components Identification
- Data Ingestion & Preprocessing:
- Scripts and modules such as data/download_data.py, preprocess_reader_data.py, and srdpr/data_helpers (which includes create_dataset.py, dataloader.py, datasets.py, and pretokenize.py).
- This layer is responsible for reading raw JSON/JSONLines input, converting it to ByteDataset format, and preparing the data for training.
- Configuration & Pipeline Management:
- The “conf” directory that contains YAML configuration files (config.yaml and pipeline-specific defaults) and “ecosystem.yaml” for possibly deployment or integration.
- The project uses Hydra for configuration (as seen by hydra-core in the environment setup).
- Model Components – Dense Encoders:
- The “dpr” directory contains submodules for the dense retriever, including models (biencoder, reader, hf_models, etc.), indexers (faiss_indexers.py), and associated utilities (tokenizers, logging, etc.).
- The “srdpr” folder contains tailored modules for SR-DPR (constants, data helpers, neural network tools like grad_cache, losses, trainer, etc.) which help implement the stratified ranking strategy.
- Training and Evaluation Orchestration:
- Training scripts such as train_dense_encoder.py, train_reader.py, and train_srdpr.py indicate different training workflows.
- Evaluation scripts like eval.py and generate_dense_embeddings.py.
- Utility Libraries:
- “libs” contains additional utility code (faiss_indexer.py, utils) that support core functionalities.
- Application Entry Points:
- Scripts such as app.py and build_index.py likely serve auxiliary functions or serve as entry points in different use cases.

##### Relationships and Interactions
- Data flows from input files (handled via the data/ directory and srdpr/data_helpers) through a preprocessing layer (converting to ByteDataset format) before being fed into various training pipelines.
- The configuration (from the conf/ directory) defines parameters for each pipeline (pos, poshard, hard, inbatch) and training jobs, determining how the training scripts “train_srdpr.py” or “train_reader.py” behave.
- The “dpr” and “srdpr” modules contain implementations of the dense encoders (supported by external libraries like PyTorch, Huggingface Transformers) and are instantiated in the training scripts.
- Model training is modularized into pipeline-specific components where different loss functions (including contrastive loss with N-pair loss) and optimization setups (using AdamW with gradient accumulation, warmup steps, etc.) come into play.
- Evaluation scripts interface with the trained models to generate dense embeddings and assess retrieval quality.

##### Architectural Patterns and Design Principles
- Modular Design: The distinct separation between data processing (data_helpers), model logic (dpr and srdpr submodules), configurations (conf/ YAML files), and execution scripts indicates a modular and pluggable design.
- Pipeline Architecture: The training is based on distinct pipelines (e.g., “pos,” “hard”) that operate with different batch sizes and loss formulations. The “regulate_factor” mechanism shows control over how these pipelines are interleaved.
- Layered Abstraction: The system distinguishes between lower-level utilities (in libs and dpr/utils) and higher-level training orchestration (in train_* scripts).

## Diagram (Mermaid)

```mermaid
flowchart TD
    Raw["Raw Data (JSON/JSONLines)"]:::data

    subgraph "Data Ingestion & Preprocessing"
        DI1["Download Data (data/download_data.py)"]:::data
        DI2["Preprocess Reader Data (preprocess_reader_data.py)"]:::data
        DI3["Data Helpers (srdpr/data_helpers/)"]:::data
        BD["ByteDataset Conversion"]:::data
    end

    subgraph "Configuration & Pipeline Management"
        CONFIG1["Configurations (conf/)"]:::config
        CONFIG2["Ecosystem Config (ecosystem.yaml)"]:::config
    end

    subgraph "Model Components - Dense Encoders"
        MODEL1["DPR Models (dpr/models/)"]:::model
        MODEL2["SR-DPR Modules (srdpr/)"]:::model
    end

    subgraph "Training and Evaluation Orchestration"
        TRAIN1["Train Dense Encoder (train_dense_encoder.py)"]:::train
        TRAIN2["Train Reader (train_reader.py)"]:::train
        TRAIN3["Train SR-DPR (train_srdpr.py)"]:::train
        EVAL1["Evaluation (eval.py)"]:::train
        EVAL2["Generate Embeddings (generate_dense_embeddings.py)"]:::train
        EVAL3["Build Index (build_index.py)"]:::train
    end

    subgraph "Utility Libraries"
        UTIL1["DPR Utils (dpr/utils/)"]:::util
        UTIL2["Libs (libs/)"]:::util
        UTIL3["SR-DPR Utils (srdpr/utils/)"]:::util
        UTIL4["General Utils (utils.py)"]:::util
    end

    %% Data Flow
    Raw --> DI1
    DI1 --> DI2
    DI2 --> DI3
    DI3 --> BD

    %% Input to Training
    BD --> TRAIN1
    BD --> TRAIN2
    BD --> TRAIN3

    %% Configuration influences model and training setup
    CONFIG1 --> MODEL1
    CONFIG2 --> MODEL1
    CONFIG1 --> TRAIN1
    CONFIG1 --> TRAIN2
    CONFIG1 --> TRAIN3
    CONFIG2 --> TRAIN1
    CONFIG2 --> TRAIN2
    CONFIG2 --> TRAIN3

    %% Model Components provide to Training pipelines
    MODEL1 --> TRAIN1
    MODEL1 --> TRAIN2
    MODEL1 --> TRAIN3
    MODEL2 --> TRAIN1
    MODEL2 --> TRAIN2
    MODEL2 --> TRAIN3

    %% Utility Libraries support training processes
    UTIL1 --- TRAIN1
    UTIL1 --- TRAIN2
    UTIL1 --- TRAIN3
    UTIL2 --- TRAIN1
    UTIL2 --- TRAIN2
    UTIL2 --- TRAIN3
    UTIL3 --- TRAIN1
    UTIL3 --- TRAIN2
    UTIL3 --- TRAIN3
    UTIL4 --- TRAIN1
    UTIL4 --- TRAIN2
    UTIL4 --- TRAIN3

    %% Training to Evaluation flow
    TRAIN3 --> EVAL1
    EVAL1 --> EVAL2
    EVAL1 --> EVAL3

    %% Style Definitions
    classDef data fill:#b3cde0,stroke:#005b96,stroke-width:2px;
    classDef config fill:#fdb462,stroke:#b2912f,stroke-width:2px;
    classDef model fill:#c7e9b4,stroke:#3182bd,stroke-width:2px;
    classDef train fill:#decbe4,stroke:#6a3d9a,stroke-width:2px;
    classDef util fill:#ffffb3,stroke:#b58800,stroke-width:2px;

    %% Click Events
    click DI1 "https://github.com/levuloihust99/sr-dpr/blob/main/data/download_data.py"
    click DI2 "https://github.com/levuloihust99/sr-dpr/blob/main/preprocess_reader_data.py"
    click DI3 "https://github.com/levuloihust99/sr-dpr/tree/main/srdpr/data_helpers/"
    click CONFIG1 "https://github.com/levuloihust99/sr-dpr/tree/main/conf/"
    click CONFIG2 "https://github.com/levuloihust99/sr-dpr/blob/main/ecosystem.yaml"
    click MODEL1 "https://github.com/levuloihust99/sr-dpr/tree/main/dpr/models/"
    click MODEL2 "https://github.com/levuloihust99/sr-dpr/tree/main/srdpr/"
    click TRAIN1 "https://github.com/levuloihust99/sr-dpr/blob/main/train_dense_encoder.py"
    click TRAIN2 "https://github.com/levuloihust99/sr-dpr/blob/main/train_reader.py"
    click TRAIN3 "https://github.com/levuloihust99/sr-dpr/blob/main/train_srdpr.py"
    click EVAL1 "https://github.com/levuloihust99/sr-dpr/blob/main/eval.py"
    click EVAL2 "https://github.com/levuloihust99/sr-dpr/blob/main/generate_dense_embeddings.py"
    click EVAL3 "https://github.com/levuloihust99/sr-dpr/blob/main/build_index.py"
    click UTIL1 "https://github.com/levuloihust99/sr-dpr/tree/main/dpr/utils/"
    click UTIL2 "https://github.com/levuloihust99/sr-dpr/tree/main/libs/"
    click UTIL3 "https://github.com/levuloihust99/sr-dpr/tree/main/srdpr/utils/"
    click UTIL4 "https://github.com/levuloihust99/sr-dpr/blob/main/utils.py"
```
