How levuloihust99/sr-dpr works
Main Components Identification
• Data Ingestion & Preprocessing:
• Scripts and modules such as data/download_data.py, preprocess_reader_data.py, and srdpr/data_helpers (which includes create_dataset.py, dataloader.py, datasets.py, and pretokenize.py).
• This layer is responsible for reading raw JSON/JSONLines input, converting it to ByteDataset format, and preparing the data for training.
• Configuration & Pipeline Management:
• The “conf” directory that contains YAML configuration files (config.yaml and pipeline-specific defaults) and “ecosystem.yaml” for possibly deployment or integration.
• The project uses Hydra for configuration (as seen by hydra-core in the environment setup).
• Model Components – Dense Encoders:
• The “dpr” directory contains submodules for the dense retriever, including models (biencoder, reader, hf_models, etc.), indexers (faiss_indexers.py), and associated utilities (tokenizers, logging, etc.).
• The “srdpr” folder contains tailored modules for SR-DPR (constants, data helpers, neural network tools like grad_cache, losses, trainer, etc.) which help implement the stratified ranking strategy.
• Training and Evaluation Orchestration:
• Training scripts such as train_dense_encoder.py, train_reader.py, and train_srdpr.py indicate different training workflows.
• Evaluation scripts like eval.py and generate_dense_embeddings.py.
• Utility Libraries:
• “libs” contains additional utility code (faiss_indexer.py, utils) that support core functionalities.
• Application Entry Points:
• Scripts such as app.py and build_index.py likely serve auxiliary functions or serve as entry points in different use cases.
Relationships and Interactions
• Data flows from input files (handled via the data/ directory and srdpr/data_helpers) through a preprocessing layer (converting to ByteDataset format) before being fed into various training pipelines.
• The configuration (from the conf/ directory) defines parameters for each pipeline (pos, poshard, hard, inbatch) and training jobs, determining how the training scripts “train_srdpr.py” or “train_reader.py” behave.
• The “dpr” and “srdpr” modules contain implementations of the dense encoders (supported by external libraries like PyTorch, Huggingface Transformers) and are instantiated in the training scripts.
• Model training is modularized into pipeline-specific components where different loss functions (including contrastive loss with N-pair loss) and optimization setups (using AdamW with gradient accumulation, warmup steps, etc.) come into play.
• Evaluation scripts interface with the trained models to generate dense embeddings and assess retrieval quality.
Architectural Patterns and Design Principles
• Modular Design: The distinct separation between data processing (data_helpers), model logic (dpr and srdpr submodules), configurations (conf/ YAML files), and execution scripts indicates a modular and pluggable design.
• Pipeline Architecture: The training is based on distinct pipelines (e.g., “pos,” “hard”) that operate with different batch sizes and loss formulations. The “regulate_factor” mechanism shows control over how these pipelines are interleaved.
• Layered Abstraction: The system distinguishes between lower-level utilities (in libs and dpr/utils) and higher-level training orchestration (in train_* scripts).