How dsba-lab/contrastive-accumulation works
Project Type and Purpose
• This repository contains a research implementation for a dense passage retriever using gradient accumulation methods under memory constraints. It is a predominantly machine learning/NLP system built around the Dense Passage Retrieval (DPR) model – adapted from nanoDPR – that supports multiple datasets (e.g., MS Marco, NQ, TREC, Trivia, WebQ).
• In addition to training the DPR model (with various resource configurations specified in YAML files), the project includes components for data downloading/preprocessing, embedding extraction, and evaluation.
Analyzing the File Structure
• Top-level folders/files:
– config: Contains YAML configuration files that define hyperparameters and training settings for different datasets and resource constraints.
– data: Scripts and notebooks for data download and preprocessing.
– src: Main source code including data handling (src/data.py), model definitions (src/model.py), training script (src/train_dpr.py), and utility modules (src/utils, src/msmarco_utils).
– scripts: Contains helper scripts for embedding extraction and testing.
– README.md: Provides high-level usage instructions and an overview of training/evaluation flows.
• This structure indicates a modular design that separates configuration, training logic, data processing, and utility scripts, which is essential when diagramming the system.
Key Components
a. Configuration Management:
– Represent the “Config Manager” component that reads YAML files under the config directory. This component supplies hyperparameters (e.g., batch sizes, gradient accumulation steps, and memory settings) to the training engine.
– Different directories inside “config” (msmarco, nq, trec, trivia, webq) indicate that the system supports multiple datasets and resource modes (vram11, vram24, etc.).
b. Data Acquisition and Preprocessing:
– Include a “Data Module” that encapsulates the scripts found in the “data” directory (including the notebook for MS Marco preprocessing and the shell script for dataset download).
– The data flows through preprocessing, then is fed into the training pipeline.
c. Model Training Engine:
– Focus on the “Training Component” (src/train_dpr.py) which is the core that ties together the data loader, model (from src/model.py), and training loop logic. This component implements the gradient accumulation mechanism (both traditional InfoNCE loss and the modified “ContAccum” scheme).
– Show the interaction with utility modules in src/utils (such as tokenizers and download utils) and specialized modules in src/msmarco_utils.
d. Embedding Extraction Module:
– Capture the embedding extraction functionality that is provided by “doc2embedding.py” and “doc2embedding_msmarco.py”, as well as the associated shell script in scripts/tools/embed.sh.
– This component outputs embeddings from passages for downstream applications.
e. Evaluation Module:
– Represent the evaluation parts of the project via “test_msmarco.py”, “test_dpr.py”, and a test shell script (scripts/tools/test.sh) that verify the model’s performance.
f. External Dependencies and Pipeline Orchestration:
– Indicate the use of external libraries such as PyTorch and Huggingface tools (e.g., Accelerate) which facilitate multi-process training.
– The command-line interface is used as the entry point to each of these modules (data preparation, training, and evaluation).
Relationships and Interactions
• The user issues commands (via CLI) which are routed to different components:
– For data preparation, the CLI triggers scripts in the “data” folder to download and preprocess datasets.
– For training, the CLI calls src/train_dpr.py including a reference to the appropriate YAML configuration file from the “config” directory.
– The training pipeline loads data using modules in “src/data.py” and “src/utils/download_data.py”, constructs the model from “src/model.py” based on hyperparameters and training configurations, computes the loss (using either traditional InfoNCE or the ContAccum variant), and periodically performs gradient accumulation and model updates.
– Post-training, embedding extraction and evaluation scripts are run to produce results.
– From the configuration files into the training engine.
– From data acquisition to preprocessing, then into the training pipeline.
– From the trained model into embedding extraction and then to evaluation.
• Use labels to denote data/configuration flows, like “YAML config”, “preprocessed data”, “model training”, “embedding output”, and “evaluation results”.
Final Reminder
• Don’t oversimplify: Even if many divisions appear granular (such as separate folders for different datasets), note them as separate configuration or operational modules connected through common training/evaluation pipelines.
• The goal is to represent the repository both as a full-stack machine learning infrastructure (data, training, embedding extraction, and evaluation) and to highlight the novel gradient accumulation methods implemented.