How akshay140601/end-to-end-nlp-system works
Project Type and Purpose
• This is an end-to-end NLP pipeline implementing a Retrieval-Augmented Generation (RAG) system for question answering about Carnegie Mellon University (CMU) and the Language Technologies Institute (LTI).
• It combines data collection (web/pdf scraping), annotation, embedding creation, retrieval (including multi-query retriever and reranker), large-language-model (LLM) answer generation, post-processing, evaluation, and significance testing.
File-Structure Components and Architectural Patterns
• Data Ingestion & Storage
• py_files_for_scraping/: scripts to scrape HTML, parse PDFs, fetch research papers
• Scraped Data/ & Scraped_pdf_file/: raw text and CSV outputs from scraping
• data/test/: gold-standard test questions and reference answers
• Annotation
• Annotations/: initial, manually/automatically generated annotations
• Final_Annotations/: cleaned, finalized question–context annotations for testing
• Embedding Generation
• Embeddings/chroma_embeddings_script.py: creates document embeddings for retrieval
• RAG Pipeline
• RAG Pipeline/rag_pipeline.py: orchestrates retrieval, reranking, and LLM calls
• RAG Pipeline/closed_book_model_script.py: baseline generation without retrieval
• Post-Processing
• Postprocessing/postprocessing.py: formats raw LLM outputs into final answer forms
• Evaluation & Reporting
• evaluation_metrics/evaluation_metrics.py: computes F1, exact-match, precision, recall
• Significance Testing/: scripts and reports comparing model variants statistically
• Reports/: Inter-Annotator Agreement (IAA) reports
• Generated answers/ & system_outputs/: model outputs for analysis
• Documentation & Contributions
• README.md, Contributions.md, report.pdf
Key Architectural Patterns
• Modular batch-pipeline (ETL) style: distinct stages for ingestion → annotation → embedding → retrieval → generation → post-processing → evaluation → reporting.
• RAG pattern with optional multi-query retriever and reranker stages.
• Separation of concerns: data collection vs. data annotation vs. model execution vs. evaluation.
a. Data Sources (leftmost)
– Web pages, PDF files, research-paper databases
b. Scraping Layer
– HTML scraper, PDF parser, Scholarly-API fetcher
→ Outputs raw text & CSV files into “Scraped Data” and “Scraped_pdf_file”
c. Annotation Layer
– annotate.py, annotate-calendar.py, annotate-csv.py
→ produces “Annotations” → cleaned into “Final_Annotations”
d. Embedding Layer
– chroma_embeddings_script.py
→ builds vector store of document embeddings
e. Retrieval Layer
– RAG Pipeline:
• Retriever (standard or multi-query) → top-k documents
• Reranker → reorders retrieved docs
f. Generation Layer
– LLAMA2-70B-chat model
• closed-book (no context)
• RAG with k contexts (various variants)
→ raw generated answers
g. Post-Processing Layer
– postprocessing.py: cleans and formats answers
h. Evaluation Layer
– evaluation_metrics.py: calculates F1, EM, precision, recall
– significance_testing_script.py: performs statistical tests
i. Reporting Layer
– Reports/ (IAA), Generated answers/, system_outputs/, report.pdf
Relationships & Data Flow
Data Sources → Scraping Layer → Raw Storage → Annotation → Final_Annotations → Embedding Layer → Vector Store → Retrieval → Reranking → LLM Generation → Post-Processing → Evaluation → Reporting.
Technologies & Frameworks to Annotate
• Python scripts (scraping, annotation, embedding, evaluation)
• Chroma (or other vector store) for embeddings
• LLAMA2-70B-chat as generation model
• Statistical packages for significance testing (e.g., SciPy or custom scripts)