# dsba-lab/contrastive-accumulation architecture

> Architecture diagram of the GitHub repository dsba-lab/contrastive-accumulation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dsba-lab/contrastive-accumulation
- Repository: https://github.com/dsba-lab/contrastive-accumulation

## Overview

##### Project Type and Purpose
- This repository contains a research implementation for a dense passage retriever using gradient accumulation methods under memory constraints. It is a predominantly machine learning/NLP system built around the Dense Passage Retrieval (DPR) model – adapted from nanoDPR – that supports multiple datasets (e.g., MS Marco, NQ, TREC, Trivia, WebQ).
- In addition to training the DPR model (with various resource configurations specified in YAML files), the project includes components for data downloading/preprocessing, embedding extraction, and evaluation.

##### Analyzing the File Structure
- Top-level folders/files:
– config: Contains YAML configuration files that define hyperparameters and training settings for different datasets and resource constraints.
– data: Scripts and notebooks for data download and preprocessing.
– src: Main source code including data handling (src/data.py), model definitions (src/model.py), training script (src/train_dpr.py), and utility modules (src/utils, src/msmarco_utils).
– scripts: Contains helper scripts for embedding extraction and testing.
– README.md: Provides high-level usage instructions and an overview of training/evaluation flows.
- This structure indicates a modular design that separates configuration, training logic, data processing, and utility scripts, which is essential when diagramming the system.

##### Key Components
a. Configuration Management:
– Represent the “Config Manager” component that reads YAML files under the config directory. This component supplies hyperparameters (e.g., batch sizes, gradient accumulation steps, and memory settings) to the training engine.
– Different directories inside “config” (msmarco, nq, trec, trivia, webq) indicate that the system supports multiple datasets and resource modes (vram11, vram24, etc.).
b. Data Acquisition and Preprocessing:
– Include a “Data Module” that encapsulates the scripts found in the “data” directory (including the notebook for MS Marco preprocessing and the shell script for dataset download).
– The data flows through preprocessing, then is fed into the training pipeline.
c. Model Training Engine:
– Focus on the “Training Component” (src/train_dpr.py) which is the core that ties together the data loader, model (from src/model.py), and training loop logic. This component implements the gradient accumulation mechanism (both traditional InfoNCE loss and the modified “ContAccum” scheme).
– Show the interaction with utility modules in src/utils (such as tokenizers and download utils) and specialized modules in src/msmarco_utils.
d. Embedding Extraction Module:
– Capture the embedding extraction functionality that is provided by “doc2embedding.py” and “doc2embedding_msmarco.py”, as well as the associated shell script in scripts/tools/embed.sh.
– This component outputs embeddings from passages for downstream applications.
e. Evaluation Module:
– Represent the evaluation parts of the project via “test_msmarco.py”, “test_dpr.py”, and a test shell script (scripts/tools/test.sh) that verify the model’s performance.
f. External Dependencies and Pipeline Orchestration:
– Indicate the use of external libraries such as PyTorch and Huggingface tools (e.g., Accelerate) which facilitate multi-process training.
– The command-line interface is used as the entry point to each of these modules (data preparation, training, and evaluation).

##### Relationships and Interactions
- The user issues commands (via CLI) which are routed to different components:
– For data preparation, the CLI triggers scripts in the “data” folder to download and preprocess datasets.
– For training, the CLI calls src/train_dpr.py including a reference to the appropriate YAML configuration file from the “config” directory.
– The training pipeline loads data using modules in “src/data.py” and “src/utils/download_data.py”, constructs the model from “src/model.py” based on hyperparameters and training configurations, computes the loss (using either traditional InfoNCE or the ContAccum variant), and periodically performs gradient accumulation and model updates.
– Post-training, embedding extraction and evaluation scripts are run to produce results.
– From the configuration files into the training engine.
– From data acquisition to preprocessing, then into the training pipeline.
– From the trained model into embedding extraction and then to evaluation.
- Use labels to denote data/configuration flows, like “YAML config”, “preprocessed data”, “model training”, “embedding output”, and “evaluation results”.

##### Final Reminder
- Don’t oversimplify: Even if many divisions appear granular (such as separate folders for different datasets), note them as separate configuration or operational modules connected through common training/evaluation pipelines.
- The goal is to represent the repository both as a full-stack machine learning infrastructure (data, training, embedding extraction, and evaluation) and to highlight the novel gradient accumulation methods implemented.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["Command-Line Interface"]:::cli
    end

    %% Application Layer
    subgraph "Application Layer"
        CONFIG["Config Manager (YAML)"]:::config

        subgraph "Data Acquisition and Preprocessing"
            DS["Download Script"]:::data
            DP["Preprocess Notebook"]:::data
        end

        TRAIN["Dense Retriever Training Engine\n(src/train_dpr.py)"]:::training

        subgraph "Embedding Extraction"
            DE1["doc2embedding.py"]:::embedding
            DE2["doc2embedding_msmarco.py"]:::embedding
            DES["Embed Shell Script"]:::embedding
        end

        subgraph "Evaluation"
            EV1["test_dpr.py"]:::evaluation
            EV2["test_msmarco.py"]:::evaluation
            EVS["Eval Shell Script"]:::evaluation
        end
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        EXT["PyTorch & Accelerate"]:::external
    end

    %% Connections from CLI to Application Components
    CLI -->|"triggers"| CONFIG
    CLI -->|"triggers"| DS
    CLI -->|"triggers"| DP
    CLI -->|"triggers"| TRAIN
    CLI -->|"triggers"| DE1
    CLI -->|"triggers"| DE2
    CLI -->|"triggers"| DES
    CLI -->|"triggers"| EV1
    CLI -->|"triggers"| EV2
    CLI -->|"triggers"| EVS

    %% Data and configuration flows within Application Layer
    CONFIG -->|"provides_config"| TRAIN
    DS -->|"provides_data"| TRAIN
    DP -->|"provides_preprocessed_data"| TRAIN
    TRAIN -->|"outputs_model"| DE1
    TRAIN -->|"outputs_model"| DE2
    TRAIN -->|"outputs_model"| DES
    DE1 -->|"produces_embeddings"| EV1
    DE2 -->|"produces_embeddings"| EV2
    DES -->|"produces_embeddings"| EVS

    %% External support
    EXT -->|"supports_training"| TRAIN

    %% Click Events
    click CONFIG "https://github.com/dsba-lab/contrastive-accumulation/tree/main/config"
    click DS "https://github.com/dsba-lab/contrastive-accumulation/blob/main/data/download_dpr_datasets.sh"
    click DP "https://github.com/dsba-lab/contrastive-accumulation/blob/main/data/msmarco_download_and_preprocess.ipynb"
    click TRAIN "https://github.com/dsba-lab/contrastive-accumulation/blob/main/src/train_dpr.py"
    click DE1 "https://github.com/dsba-lab/contrastive-accumulation/blob/main/doc2embedding.py"
    click DE2 "https://github.com/dsba-lab/contrastive-accumulation/blob/main/doc2embedding_msmarco.py"
    click DES "https://github.com/dsba-lab/contrastive-accumulation/blob/main/scripts/tools/embed.sh"
    click EV1 "https://github.com/dsba-lab/contrastive-accumulation/blob/main/test_dpr.py"
    click EV2 "https://github.com/dsba-lab/contrastive-accumulation/blob/main/test_msmarco.py"
    click EVS "https://github.com/dsba-lab/contrastive-accumulation/blob/main/scripts/tools/test.sh"

    %% Styles
    classDef cli fill:#f7d6c1,stroke:#333,stroke-width:2px;
    classDef config fill:#d1e8ff,stroke:#333,stroke-width:2px;
    classDef data fill:#d9f5d9,stroke:#333,stroke-width:2px;
    classDef training fill:#fff2cc,stroke:#333,stroke-width:2px;
    classDef embedding fill:#e2d1f9,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#ffd1dc,stroke:#333,stroke-width:2px;
    classDef external fill:#c9daf8,stroke:#333,stroke-width:2px;
```
