How mayer123/udt-qa works
• DPR Module: This is the core retrieval engine. It includes subdirectories for configurations (under DPR/conf), data preprocessing, model definitions (biencoder, reader), indexers (FAISS for embedding indexing), training scripts (train_dense_encoder.py, train_extractive_reader.py), and inference scripts (dense_retriever.py, generate_dense_embeddings.py).
• FiE_reader and fid_reader Modules: These modules focus on training and evaluating reader models. They work on processing the retrieved passages, extracting the final answers, and perhaps evaluating retrieval quality.
• Chainer Module: Contains scripts for chaining, re-ranking, or passing retrieval outputs to subsequent stages (e.g., rerank_passages.py, run_chainer.py, run_rerank.sh).
• Verbalizer Module: Focused on natural language verbalization of structured knowledge. It contains utilities for data processing, generation (generate.sh), post-processing, and training (train.sh) using frameworks like PyTorch Lightning.
• Data Download & Preparation Utilities: Scripts like download_data.py and download_data_hf.py along with configuration files that manage different resource keys for models and data.
Relationships and data flows
• Data Ingestion: The system starts by downloading and preparing the raw and structured knowledge (tables, KBs) using the provided download scripts.
• Retrieval: The DPR module takes in these knowledge resources, processes them via its encoder models and indexing components, and performs dense retrieval to get relevant passages.
• Passage Processing: The outputs from retrieval can be sent to the FiE_reader (or fid_reader) modules, where reader models process the passages to extract answers.
• Post-Processing & Chaining: The Chainer components use the outputs (e.g., candidate passages or answer spans) to perform re-ranking or chain-of-thought reasoning.
• Verbalization: In parallel or afterward, the Verbalizer module can take in structured data and transform it into a verbal (natural language) format, which may then be used either for enhanced retrieval or as additional input for answer generation.
Important architectural patterns
• Modularity: Each functional area (retrieval, reading, chaining, verbalization) is encapsulated in its own directory with clearly separated configuration files and scripts.
• Pipeline Structure: Data flows from download and pre-processing to retrieval, then to reading/answer extraction, and finally to output re-ranking and post-processing.
• Extensibility: By separating configurations (e.g., different YAML configuration files under DPR/conf) and managing multiple environments (one for DPR and one for Verbalizer), the project facilitates independent development and updates.
• Distributed and Parallel Processing: Several scripts (e.g., generate_dense_embeddings.py, the use of torch.distributed.launch) indicate a design intended for running processes in parallel, particularly when handling large datasets.
Relevant technologies and frameworks
• Python and Linux shell scripts.
• PyTorch along with torch.distributed for training and embedding generation.
• HuggingFace Transformers for model handling, especially in the verbalizer and possibly in some reader models.
• Additional libraries for specialized indexing (e.g., FAISS) and for data processing.
• PyTorch Lightning (in Verbalizer) to simplify training and model management.
Detail the design with as much granularity as possible
• Configuration files (especially those under DPR/conf) and show their connections to their respective training/inference modules.
• Separation of concerns in the codebase: retrieval (DPR), reader (FiE_reader and fid_reader), chaining (Chainer), and verbalization (Verbalizer).
• Notes on distributed training aspects (e.g., torch.distributed.launch on train_dense_encoder.py) as an annotation or side note.