How rishichhabra29/a-domain-specific-end-to-end-qa-system-using-dense-retrieval-and-prompt-aware-generation works
Project type and purpose
• This is an open-source, domain-specific Machine Learning pipeline/tool: an end-to-end Question Answering system for electronics product reviews.
• It combines two decoupled modules—a Dense Passage Retriever (DPR based on RoBERTa) and a Prompt-Aware Generator (FLAN-T5)—to fetch relevant review snippets and generate fluent answers.
File structure
• Top-level scripts:
• data_preprocessing.py – cleans and splits raw SubjQA CSV files
• train_dpr.py / test_dpr.py – train and validate the retriever
• train_generator.py / test_generator.py – train and validate the generator
• qa_pipeline.py – loads models, precomputes embeddings, orchestrates retrieval→generation
• qa_eval.py – runs end-to-end evaluation (retriever + generator) and computes BERTScore
• qa_app.py – example inference application (CLI or API wrapper)
• Directories:
• datasets/ – raw and split CSVs (subjectqa_train/val/test)
• outputs/ – model outputs, metrics, cached embeddings
• Configuration:
• requirements.txt – library dependencies
Extract architectural insights from README
• Modular Retriever–Generator architecture
• Data flow: raw reviews → preprocessing → training splits → model training → saved artifacts → inference/evaluation
• Training details: contrastive learning with dynamic hard negatives for DPR; prompt-aware FLAN-T5 fine-tuning for generator
• Inference pipeline: encode question, retrieve top-k passages, generate answer from each, select or aggregate
Main components
A. Data Ingestion & Preprocessing
• Input: subjectqa_full.csv
• Output: train/val/test CSVs
B. Retriever Module
• train_dpr.py → DPR encoder (RoBERTa-base) + tokenizer → saved in custom_dpr/
• test_dpr.py (unit-test)
C. Generator Module
• train_generator.py → FLAN-T5 fine-tuned → saved in custom_generator/
• test_generator.py (unit-test)
D. Inference Pipeline (qa_pipeline.py & qa_app.py)
1. Load DPR + precompute passage embeddings (FAISS or in-memory index)
2. Encode user question → retrieve top-k contexts
3. For each context: format prompt → run generator → obtain answer
4. Aggregate or pick top answer
E. Evaluation Module (qa_eval.py)
• Uses test set → full pipeline → computes BERTScore metrics
F. Outputs & Artifacts
• Model directories (custom_dpr/, custom_generator/)
• Cached embeddings, logs, evaluation results in outputs/
Relationships and interactions
• Data flows from datasets/ → preprocessing → training scripts
• Training scripts produce model artifacts → used by pipeline & evaluation
• qa_app.py wraps the pipeline for ad hoc queries
• test_*.py scripts validate individual components
• qa_eval.py orchestrates full-system testing and metrics calculation
Architectural patterns and design principles
• Modular design: retriever and generator decoupled, interchangeable
• Pipeline pattern: clear separation between data preparation, training, inference, evaluation
• Contrastive learning pattern in retriever (dynamic hard negative mining)
• Prompt-aware generation: consistent “question: … context: …” format
• Cache vs compute tradeoff: precompute passage embeddings for fast retrieval
Key technologies, frameworks, and libraries
• PyTorch, Hugging Face Transformers (RoBERTa, FLAN-T5)
• Datasets & pandas for CSV handling
• scikit-learn for splits, evaluate library for BERTScore
• FAISS or custom in-memory index for nearest-neighbor retrieval
• tqdm for progress bars, NumPy for arrays