# microsoft/ance architecture

> Architecture diagram of the GitHub repository microsoft/ance, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/ance
- Repository: https://github.com/microsoft/ance

## Overview

##### Project Type and Purpose
- This is a research-grade machine-learning pipeline for dense text retrieval, not a web or full-stack app.
- It implements ANCE (Approximate Nearest Neighbor Contrastive Learning) training loops: data preparation → warmup training → iterative ANN data generation + model training → inference → evaluation.

##### File Structure Analysis
- commands/ (bash scripts): orchestrate end-to-end steps (download, preprocess, train, inference).
- data/ (Python): raw data preprocessing (MSMARCO, DPR).
- drivers/ (Python entrypoints): run warmup, ANCE data generation, ANCE training, DPR variants.
- model/SEED_Encoder/ + model/models.py: model implementation (tokenizer, transformer, encoder configuration).
- evaluation/ (Jupyter): ranking metrics calculation.
- utils/: shared utilities (optimizer, DPR helpers, evaluation metrics).
- setup.py & packaging: installs the project as a Python package.

##### Key Components
a. Raw Data Sources
- MSMARCO corpus, DPR OpenQA data (external storage or S3 buckets)
b. Data Processing Module
- data/msmarco_data.py, data/DPR_data.py → outputs tokenized, chunked examples
c. Warmup Trainer
- drivers/run_warmup.py (BM25-based warmup BERT checkpoint)
- produces initial model checkpoint
d. ANN Data Generation Service
- drivers/run_ann_data_gen.py & run_ann_data_gen_dpr.py
- builds/updates ANN index (e.g., Faiss) using current model
- outputs negative-sampled training data
e. ANCE Training Service
- drivers/run_ann.py & run_ann_dpr.py
- consumes preprocessed data + ANN negatives → updates model checkpoints
f. Iterative Loop
- parallel jobs: ANN Data Generation ↔ ANCE Training feed into each other
g. Inference Module
- drivers/run_ann_data_gen.py with --inference flag or commands/run_inference.sh
- computes embeddings for queries and documents
h. Evaluation Module
- evaluation/Calculate Metrics.ipynb → computes MRR, NDCG, recall

##### Relationships and Data Flow
- From Initial Checkpoint → ANN Data Generation → ANN Negatives
- From ANN Negatives + Preprocessed Data → ANCE Training → New Checkpoint

##### Architectural Patterns and Principles
- Modular CLI-driven services (drivers + commands)
- Iterative data-model feedback loop (CESS style)
- Separation of concerns: preprocessing, index building, training, inference, evaluation
- Use of standard packaging (setup.py) for reproducibility
- Parallel processing: training and data generation run concurrently

##### Technologies and Frameworks to Highlight
- PyTorch (torch.distributed.launch)
- FAISS or any ANN index library (implicit)
- Transformer models (BERT-Siamese via model/SEED_Encoder)
- Python CLI modules (argparse, driver scripts)
- Shell orchestration (bash scripts in commands/)
- Jupyter notebook for metrics

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Layer
    subgraph "Data Preparation"
        RawData1["MSMARCO Corpus"]:::datastore
        RawData2["DPR OpenQA Data"]:::datastore
        Process1["msmarco_data.py"]:::compute
        Process2["DPR_data.py"]:::compute
        Process3["process_fn.py"]:::compute
        Preprocessed["Preprocessed Data"]:::datastore
    end

    RawData1 -->|input| Process1
    RawData2 -->|input| Process2
    Process1 --> Process3
    Process2 --> Process3
    Process3 -->|outputs| Preprocessed

    %% Warmup Training
    subgraph "Warmup Training"
        WarmupScript["run_train_warmup.sh"]:::orchestration
        WarmupDriver["run_warmup.py"]:::compute
        WarmupChk["warmup_checkpoint.pt"]:::artifact
    end

    Preprocessed -->|uses| WarmupDriver
    WarmupScript --> WarmupDriver
    WarmupDriver --> WarmupChk

    %% Iterative ANN-ANCE Training Loop
    subgraph "Training Loop"
        direction TB
        subgraph "ANN Data Generation"
            ANNGenScript["run_ann_data_gen.sh"]:::orchestration
            ANNGenDriver["run_ann_data_gen.py"]:::compute
            ANNGenDPRScript["run_ann_data_gen_dpr.sh"]:::orchestration
            ANNGenDPRDriver["run_ann_data_gen_dpr.py"]:::compute
            NegSamples["ANN Negatives"]:::datastore
        end
        subgraph "ANCE Training"
            TrainScript["run_train.sh"]:::orchestration
            TrainDriver["run_ann.py"]:::compute
            TrainDPRScript["run_train_dpr.sh"]:::orchestration
            TrainDPRDriver["run_ann_dpr.py"]:::compute
            NewChk["ann_checkpoint.pt"]:::artifact
        end
    end

    WarmupChk -->|initial model| ANNGenDriver
    WarmupChk --> ANNGenDPRDriver
    ANNGenScript --> ANNGenDriver
    ANNGenDPRScript --> ANNGenDPRDriver
    ANNGenDriver --> NegSamples
    ANNGenDPRDriver --> NegSamples

    Preprocessed -->|data| TrainDriver
    NegSamples --> TrainDriver
    TrainScript --> TrainDriver
    TrainDPRScript --> TrainDPRDriver
    NegSamples --> TrainDPRDriver
    Preprocessed --> TrainDPRDriver
    TrainDriver --> NewChk
    TrainDPRDriver --> NewChk

    NewChk -->|feedback| ANNGenDriver
    NewChk --> ANNGenDPRDriver

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        InferenceScript["run_inference.sh"]:::orchestration
        InferenceDriver["run_ann_data_gen.py"]:::compute
        Embeddings["Embeddings"]:::datastore
        EvalNotebook["Calculate Metrics.ipynb"]:::compute
        EvalReport["Metrics Report"]:::artifact
    end

    NewChk --> InferenceDriver
    InferenceScript --> InferenceDriver
    InferenceDriver --> Embeddings
    Embeddings --> EvalNotebook
    EvalNotebook --> EvalReport

    %% Model Implementation
    subgraph "Model Implementation"
        ModelFile["models.py"]:::compute
        subgraph "SEED_Encoder/"
            EncoderFiles["tokenizer & transformer configs"]:::compute
        end
    end

    ModelFile
    EncoderFiles

    %% Shared Utilities
    subgraph "Shared Utils"
        Util1["dpr_utils.py"]:::compute
        Util2["lamb.py"]:::compute
        Util3["util.py"]:::compute
        Util4["eval_mrr.py"]:::compute
        Util5["msmarco_eval.py"]:::compute
    end

    %% Click Events
    click Process1 "https://github.com/microsoft/ance/blob/master/data/msmarco_data.py"
    click Process2 "https://github.com/microsoft/ance/blob/master/data/DPR_data.py"
    click Process3 "https://github.com/microsoft/ance/blob/master/data/process_fn.py"
    click WarmupScript "https://github.com/microsoft/ance/blob/master/commands/run_train_warmup.sh"
    click WarmupDriver "https://github.com/microsoft/ance/blob/master/drivers/run_warmup.py"
    click ANNGenScript "https://github.com/microsoft/ance/blob/master/commands/run_ann_data_gen.sh"
    click ANNGenDriver "https://github.com/microsoft/ance/blob/master/drivers/run_ann_data_gen.py"
    click ANNGenDPRScript "https://github.com/microsoft/ance/blob/master/commands/run_ann_data_gen_dpr.sh"
    click ANNGenDPRDriver "https://github.com/microsoft/ance/blob/master/drivers/run_ann_data_gen_dpr.py"
    click TrainScript "https://github.com/microsoft/ance/blob/master/commands/run_train.sh"
    click TrainDriver "https://github.com/microsoft/ance/blob/master/drivers/run_ann.py"
    click TrainDPRScript "https://github.com/microsoft/ance/blob/master/commands/run_train_dpr.sh"
    click TrainDPRDriver "https://github.com/microsoft/ance/blob/master/drivers/run_ann_dpr.py"
    click InferenceScript "https://github.com/microsoft/ance/blob/master/commands/run_inference.sh"
    click InferenceDriver "https://github.com/microsoft/ance/blob/master/drivers/run_ann_data_gen.py"
    click EvalNotebook "https://github.com/microsoft/ance/blob/master/evaluation/Calculate Metrics.ipynb"
    click Util4 "https://github.com/microsoft/ance/blob/master/utils/eval_mrr.py"
    click Util5 "https://github.com/microsoft/ance/blob/master/utils/msmarco_eval.py"
    click ModelFile "https://github.com/microsoft/ance/blob/master/model/models.py"
    click EncoderFiles "https://github.com/microsoft/ance/tree/master/model/SEED_Encoder/"

    %% Styles
    classDef datastore fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px
    classDef artifact fill:#FFFFE0,stroke:#333,stroke-width:1px
    classDef orchestration fill:#FFA500,stroke:#333,stroke-width:1px
```
