How facebookresearch/fid works
Project type and purpose
• This is an open-source question-answering toolkit that implements two key pipelines:
• Fusion-in-Decoder “reader” models (generative QA)
• Retriever training via distillation from the reader
• It provides end-to-end scripts to download raw QA datasets (NaturalQuestions, TriviaQA), preprocess them, index a knowledge source (Wikipedia), train and evaluate both retrievers and readers, and distill reader knowledge back into the retriever.
File structure
• Top-level scripts:
• get-data.sh, get-model.sh – downloading datasets & pretrained models
• train_reader.py, test_reader.py – train/evaluate the Fusion-in-Decoder reader
• train_retriever.py – train the retriever using distillation
• generate_passage_embeddings.py, passage_retrieval.py – index passages and perform retrieval at query time
• evaluate_retrieved_passages.py – analysis utilities
• src/ directory (core library modules):
• data.py – data loading & formatting
• preprocess.py – data preprocessing (tokenization, truncation)
• index.py – embedding generation and indexing logic
• model.py – model definitions (reader & retriever architectures)
• evaluation.py – metrics & evaluation loops
• options.py – CLI argument parsing
• util.py, slurm.py – helper functions, cluster job wrappers
• Configs and environments: requirements.txt, setup.py
README for additional insights
• Two main workflows are described:
I. Fusion-in-Decoder reader train/test
II. Distilling knowledge from reader to retriever (4 steps: cross-attention score extraction, retriever training, indexing, retrieval)
• Calls out core dependencies: PyTorch, Transformers, NumPy
• Mentions “inspired by DPR” – so embedding + FAISS-style index concept
Elements
• Blue for data components
• Green for training pipelines
• Orange for inference / indexing pipelines
• Gray for external dependencies (PyTorch, Transformers, FAISS)
Detail without overthinking
• Break the system into as many logical pieces as possible
• Both offline (training / indexing) and online (retrieval → reader) flows