How microsoft/ance works
Project Type and Purpose
• This is a research-grade machine-learning pipeline for dense text retrieval, not a web or full-stack app.
• It implements ANCE (Approximate Nearest Neighbor Contrastive Learning) training loops: data preparation → warmup training → iterative ANN data generation + model training → inference → evaluation.
File Structure Analysis
• commands/ (bash scripts): orchestrate end-to-end steps (download, preprocess, train, inference).
• data/ (Python): raw data preprocessing (MSMARCO, DPR).
• drivers/ (Python entrypoints): run warmup, ANCE data generation, ANCE training, DPR variants.
• model/SEED_Encoder/ + model/models.py: model implementation (tokenizer, transformer, encoder configuration).
• evaluation/ (Jupyter): ranking metrics calculation.
• utils/: shared utilities (optimizer, DPR helpers, evaluation metrics).
• setup.py & packaging: installs the project as a Python package.
Key Components
a. Raw Data Sources
• MSMARCO corpus, DPR OpenQA data (external storage or S3 buckets)
b. Data Processing Module
• data/msmarco_data.py, data/DPR_data.py → outputs tokenized, chunked examples
c. Warmup Trainer
• drivers/run_warmup.py (BM25-based warmup BERT checkpoint)
• produces initial model checkpoint
d. ANN Data Generation Service
• drivers/run_ann_data_gen.py & run_ann_data_gen_dpr.py
• builds/updates ANN index (e.g., Faiss) using current model
• outputs negative-sampled training data
e. ANCE Training Service
• drivers/run_ann.py & run_ann_dpr.py
• consumes preprocessed data + ANN negatives → updates model checkpoints
f. Iterative Loop
• parallel jobs: ANN Data Generation ↔ ANCE Training feed into each other
g. Inference Module
• drivers/run_ann_data_gen.py with --inference flag or commands/run_inference.sh
• computes embeddings for queries and documents
h. Evaluation Module
• evaluation/Calculate Metrics.ipynb → computes MRR, NDCG, recall
Relationships and Data Flow
• From Initial Checkpoint → ANN Data Generation → ANN Negatives
• From ANN Negatives + Preprocessed Data → ANCE Training → New Checkpoint
Architectural Patterns and Principles
• Modular CLI-driven services (drivers + commands)
• Iterative data-model feedback loop (CESS style)
• Separation of concerns: preprocessing, index building, training, inference, evaluation
• Use of standard packaging (setup.py) for reproducibility
• Parallel processing: training and data generation run concurrently
Technologies and Frameworks to Highlight
• PyTorch (torch.distributed.launch)
• FAISS or any ANN index library (implicit)
• Transformer models (BERT-Siamese via model/SEED_Encoder)
• Python CLI modules (argparse, driver scripts)
• Shell orchestration (bash scripts in commands/)
• Jupyter notebook for metrics