# rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation architecture

> Architecture diagram of the GitHub repository rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation
- Repository: https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation

## Overview

##### Project type and purpose
- This is an open-source, domain-specific Machine Learning pipeline/tool: an end-to-end Question Answering system for electronics product reviews.
- It combines two decoupled modules—a Dense Passage Retriever (DPR based on RoBERTa) and a Prompt-Aware Generator (FLAN-T5)—to fetch relevant review snippets and generate fluent answers.

##### File structure
- Top-level scripts:
- data_preprocessing.py – cleans and splits raw SubjQA CSV files
- train_dpr.py / test_dpr.py – train and validate the retriever
- train_generator.py / test_generator.py – train and validate the generator
- qa_pipeline.py – loads models, precomputes embeddings, orchestrates retrieval→generation
- qa_eval.py – runs end-to-end evaluation (retriever + generator) and computes BERTScore
- qa_app.py – example inference application (CLI or API wrapper)
- Directories:
- datasets/ – raw and split CSVs (subjectqa_train/val/test)
- outputs/ – model outputs, metrics, cached embeddings
- Configuration:
- requirements.txt – library dependencies

##### Extract architectural insights from README
- Modular Retriever–Generator architecture
- Data flow: raw reviews → preprocessing → training splits → model training → saved artifacts → inference/evaluation
- Training details: contrastive learning with dynamic hard negatives for DPR; prompt-aware FLAN-T5 fine-tuning for generator
- Inference pipeline: encode question, retrieve top-k passages, generate answer from each, select or aggregate

##### Main components
A. Data Ingestion & Preprocessing
- Input: subjectqa_full.csv
- Output: train/val/test CSVs
B. Retriever Module
- train_dpr.py → DPR encoder (RoBERTa-base) + tokenizer → saved in custom_dpr/
- test_dpr.py (unit-test)
C. Generator Module
- train_generator.py → FLAN-T5 fine-tuned → saved in custom_generator/
- test_generator.py (unit-test)
D. Inference Pipeline (qa_pipeline.py & qa_app.py)
1. Load DPR + precompute passage embeddings (FAISS or in-memory index)
2. Encode user question → retrieve top-k contexts
3. For each context: format prompt → run generator → obtain answer
4. Aggregate or pick top answer
E. Evaluation Module (qa_eval.py)
- Uses test set → full pipeline → computes BERTScore metrics
F. Outputs & Artifacts
- Model directories (custom_dpr/, custom_generator/)
- Cached embeddings, logs, evaluation results in outputs/

##### Relationships and interactions
- Data flows from datasets/ → preprocessing → training scripts
- Training scripts produce model artifacts → used by pipeline & evaluation
- qa_app.py wraps the pipeline for ad hoc queries
- test_*.py scripts validate individual components
- qa_eval.py orchestrates full-system testing and metrics calculation

##### Architectural patterns and design principles
- Modular design: retriever and generator decoupled, interchangeable
- Pipeline pattern: clear separation between data preparation, training, inference, evaluation
- Contrastive learning pattern in retriever (dynamic hard negative mining)
- Prompt-aware generation: consistent “question: … context: …” format
- Cache vs compute tradeoff: precompute passage embeddings for fast retrieval

##### Key technologies, frameworks, and libraries
- PyTorch, Hugging Face Transformers (RoBERTa, FLAN-T5)
- Datasets & pandas for CSV handling
- scikit-learn for splits, evaluate library for BERTScore
- FAISS or custom in-memory index for nearest-neighbor retrieval
- tqdm for progress bars, NumPy for arrays

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Preparation"
        DS_RAW["datasets/ (subjectqa_full.csv)"]:::storage
        DP["data_preprocessing.py"]:::blue
        DS_SPLIT["datasets/ (train/val/test CSVs)"]:::storage
        DS_RAW --> DP --> DS_SPLIT
    end

    subgraph "Training Modules"
        TDPR["train_dpr.py"]:::green
        TSDPR["test_dpr.py"]:::purple
        TGEN["train_generator.py"]:::green
        TSGEN["test_generator.py"]:::purple
        MDPR["custom_dpr/"]:::storage
        MGEN["custom_generator/"]:::storage

        DS_SPLIT --> TDPR --> MDPR --> TSDPR
        DS_SPLIT --> TGEN --> MGEN --> TSGEN
    end

    subgraph "Inference Pipeline"
        PIPE["qa_pipeline.py"]:::orange
        IDX["Passage Embeddings Index (FAISS/in-memory)"]:::index
        APP["qa_app.py"]:::orange

        MDPR --> PIPE
        MGEN --> PIPE
        PIPE --> IDX
        APP --> PIPE
    end

    subgraph "Evaluation & Testing"
        DS_TEST["datasets/ (test CSV)"]:::storage
        EVAL["qa_eval.py"]:::purple
        OUT["outputs/ (metrics, cached embeddings)"]:::storage

        DS_TEST --> EVAL --> OUT
    end

    subgraph "Configuration"
        CFG["requirements.txt"]:::blue
        CFG --> DP
        CFG --> TDPR
        CFG --> TGEN
        CFG --> PIPE
        CFG --> EVAL
    end

    click DP "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/data_preprocessing.py"
    click DS_RAW "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/tree/main/datasets/"
    click DS_SPLIT "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/tree/main/datasets/"
    click TDPR "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/train_dpr.py"
    click TSDPR "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/test_dpr.py"
    click TGEN "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/train_generator.py"
    click TSGEN "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/test_generator.py"
    click PIPE "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/qa_pipeline.py"
    click APP "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/qa_app.py"
    click EVAL "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/qa_eval.py"
    click OUT "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/tree/main/outputs/"
    click CFG "https://github.com/rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation/blob/main/requirements.txt"

    classDef blue fill:#D0E8FF,stroke:#003366;
    classDef green fill:#DFFFD0,stroke:#336600;
    classDef orange fill:#FFE8CC,stroke:#994C00;
    classDef purple fill:#E8D0FF,stroke:#660099;
    classDef storage fill:#FFF2CC,stroke:#666600,stroke-dasharray: 5 5;
    classDef index fill:#FFECB3,stroke:#CC7A00,stroke-dasharray: 2 2;
```
