# mayer123/udt-qa architecture

> Architecture diagram of the GitHub repository mayer123/udt-qa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mayer123/udt-qa
- Repository: https://github.com/mayer123/udt-qa

## Overview

- DPR Module: This is the core retrieval engine. It includes subdirectories for configurations (under DPR/conf), data preprocessing, model definitions (biencoder, reader), indexers (FAISS for embedding indexing), training scripts (train_dense_encoder.py, train_extractive_reader.py), and inference scripts (dense_retriever.py, generate_dense_embeddings.py).
- FiE_reader and fid_reader Modules: These modules focus on training and evaluating reader models. They work on processing the retrieved passages, extracting the final answers, and perhaps evaluating retrieval quality.
- Chainer Module: Contains scripts for chaining, re-ranking, or passing retrieval outputs to subsequent stages (e.g., rerank_passages.py, run_chainer.py, run_rerank.sh).
- Verbalizer Module: Focused on natural language verbalization of structured knowledge. It contains utilities for data processing, generation (generate.sh), post-processing, and training (train.sh) using frameworks like PyTorch Lightning.
- Data Download & Preparation Utilities: Scripts like download_data.py and download_data_hf.py along with configuration files that manage different resource keys for models and data.

##### Relationships and data flows
- Data Ingestion: The system starts by downloading and preparing the raw and structured knowledge (tables, KBs) using the provided download scripts.
- Retrieval: The DPR module takes in these knowledge resources, processes them via its encoder models and indexing components, and performs dense retrieval to get relevant passages.
- Passage Processing: The outputs from retrieval can be sent to the FiE_reader (or fid_reader) modules, where reader models process the passages to extract answers.
- Post-Processing & Chaining: The Chainer components use the outputs (e.g., candidate passages or answer spans) to perform re-ranking or chain-of-thought reasoning.
- Verbalization: In parallel or afterward, the Verbalizer module can take in structured data and transform it into a verbal (natural language) format, which may then be used either for enhanced retrieval or as additional input for answer generation.

##### Important architectural patterns
- Modularity: Each functional area (retrieval, reading, chaining, verbalization) is encapsulated in its own directory with clearly separated configuration files and scripts.
- Pipeline Structure: Data flows from download and pre-processing to retrieval, then to reading/answer extraction, and finally to output re-ranking and post-processing.
- Extensibility: By separating configurations (e.g., different YAML configuration files under DPR/conf) and managing multiple environments (one for DPR and one for Verbalizer), the project facilitates independent development and updates.
- Distributed and Parallel Processing: Several scripts (e.g., generate_dense_embeddings.py, the use of torch.distributed.launch) indicate a design intended for running processes in parallel, particularly when handling large datasets.

##### Relevant technologies and frameworks
- Python and Linux shell scripts.
- PyTorch along with torch.distributed for training and embedding generation.
- HuggingFace Transformers for model handling, especially in the verbalizer and possibly in some reader models.
- Additional libraries for specialized indexing (e.g., FAISS) and for data processing.
- PyTorch Lightning (in Verbalizer) to simplify training and model management.

##### Detail the design with as much granularity as possible
- Configuration files (especially those under DPR/conf) and show their connections to their respective training/inference modules.
- Separation of concerns in the codebase: retrieval (DPR), reader (FiE_reader and fid_reader), chaining (Chainer), and verbalization (Verbalizer).
- Notes on distributed training aspects (e.g., torch.distributed.launch on train_dense_encoder.py) as an annotation or side note.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DataPrep["Data Download & Preparation Utilities"]:::data
    DPR["DPR Module<br>- Configurations (YAML)<br>- Data Preprocessing<br>- Encoder Models<br>- Indexers (FAISS)<br>- Training Scripts<br>- Inference Scripts"]:::dpr
    FiE["FiE_reader Module<br>- fie_model.py<br>- train_qa_fie.py"]:::reader
    fid["fid_reader Module<br>- train_reader.py<br>- src/"]:::reader
    Chainer["Chainer Module<br>- run_chainer.py<br>- rerank_passages.py<br>- run_rerank.sh"]:::chain
    Verbalizer["Verbalizer Module<br>- generate.sh<br>- train.sh<br>- data/"]:::verbal
    Answer["Answer Aggregation"]:::answer

    %% Data flows
    DataPrep -->|"ingests"| DPR
    DataPrep -->|"supplies"| Verbalizer
    DPR -->|"retrieves"| FiE
    DPR -->|"retrieves"| fid
    FiE -->|"sends_output"| Chainer
    fid -->|"sends_output"| Chainer
    Verbalizer -->|"augments"| Chainer
    Chainer -->|"aggregates"| Answer

    %% Click Events
    click DataPrep "https://github.com/mayer123/udt-qa/blob/main/DPR/dpr/data/download_data.py"
    click DPR "https://github.com/mayer123/udt-qa/tree/main/DPR/"
    click FiE "https://github.com/mayer123/udt-qa/tree/main/FiE_reader/"
    click fid "https://github.com/mayer123/udt-qa/tree/main/fid_reader/"
    click Chainer "https://github.com/mayer123/udt-qa/tree/main/Chainer/"
    click Verbalizer "https://github.com/mayer123/udt-qa/tree/main/Verbalizer/"

    %% Styles
    classDef data fill:#ffe6cc,stroke:#333,stroke-width:2px;
    classDef dpr fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef reader fill:#d5f5e3,stroke:#333,stroke-width:2px;
    classDef chain fill:#f5b7b1,stroke:#333,stroke-width:2px;
    classDef verbal fill:#fcf3cf,stroke:#333,stroke-width:2px;
    classDef answer fill:#d2b4de,stroke:#333,stroke-width:2px;
```
