# akshay140601/end-to-end-nlp-system architecture

> Architecture diagram of the GitHub repository akshay140601/end-to-end-nlp-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akshay140601/end-to-end-nlp-system
- Repository: https://github.com/akshay140601/end-to-end-nlp-system

## Overview

##### Project Type and Purpose
- This is an end-to-end NLP pipeline implementing a Retrieval-Augmented Generation (RAG) system for question answering about Carnegie Mellon University (CMU) and the Language Technologies Institute (LTI).
- It combines data collection (web/pdf scraping), annotation, embedding creation, retrieval (including multi-query retriever and reranker), large-language-model (LLM) answer generation, post-processing, evaluation, and significance testing.

##### File-Structure Components and Architectural Patterns
- Data Ingestion & Storage
- py_files_for_scraping/: scripts to scrape HTML, parse PDFs, fetch research papers
- Scraped Data/ & Scraped_pdf_file/: raw text and CSV outputs from scraping
- data/test/: gold-standard test questions and reference answers
- Annotation
- Annotations/: initial, manually/automatically generated annotations
- Final_Annotations/: cleaned, finalized question–context annotations for testing
- Embedding Generation
- Embeddings/chroma_embeddings_script.py: creates document embeddings for retrieval
- RAG Pipeline
- RAG Pipeline/rag_pipeline.py: orchestrates retrieval, reranking, and LLM calls
- RAG Pipeline/closed_book_model_script.py: baseline generation without retrieval
- Post-Processing
- Postprocessing/postprocessing.py: formats raw LLM outputs into final answer forms
- Evaluation & Reporting
- evaluation_metrics/evaluation_metrics.py: computes F1, exact-match, precision, recall
- Significance Testing/: scripts and reports comparing model variants statistically
- Reports/: Inter-Annotator Agreement (IAA) reports
- Generated answers/ & system_outputs/: model outputs for analysis
- Documentation & Contributions
- README.md, Contributions.md, report.pdf

##### Key Architectural Patterns
- Modular batch-pipeline (ETL) style: distinct stages for ingestion → annotation → embedding → retrieval → generation → post-processing → evaluation → reporting.
- RAG pattern with optional multi-query retriever and reranker stages.
- Separation of concerns: data collection vs. data annotation vs. model execution vs. evaluation.

a. Data Sources (leftmost)
– Web pages, PDF files, research-paper databases
b. Scraping Layer
– HTML scraper, PDF parser, Scholarly-API fetcher
→ Outputs raw text & CSV files into “Scraped Data” and “Scraped_pdf_file”
c. Annotation Layer
– annotate.py, annotate-calendar.py, annotate-csv.py
→ produces “Annotations” → cleaned into “Final_Annotations”
d. Embedding Layer
– chroma_embeddings_script.py
→ builds vector store of document embeddings
e. Retrieval Layer
– RAG Pipeline:
- Retriever (standard or multi-query) → top-k documents
- Reranker → reorders retrieved docs
f. Generation Layer
– LLAMA2-70B-chat model
- closed-book (no context)
- RAG with k contexts (various variants)
→ raw generated answers
g. Post-Processing Layer
– postprocessing.py: cleans and formats answers
h. Evaluation Layer
– evaluation_metrics.py: calculates F1, EM, precision, recall
– significance_testing_script.py: performs statistical tests
i. Reporting Layer
– Reports/ (IAA), Generated answers/, system_outputs/, report.pdf

##### Relationships & Data Flow
Data Sources → Scraping Layer → Raw Storage → Annotation → Final_Annotations → Embedding Layer → Vector Store → Retrieval → Reranking → LLM Generation → Post-Processing → Evaluation → Reporting.

##### Technologies & Frameworks to Annotate
- Python scripts (scraping, annotation, embedding, evaluation)
- Chroma (or other vector store) for embeddings
- LLAMA2-70B-chat as generation model
- Statistical packages for significance testing (e.g., SciPy or custom scripts)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Prep
    subgraph "Data Prep"
        direction TB
        DS["Data Sources\n(Web, PDF, Research DB)"]:::external
        SS["Scraping Scripts"]:::ingestion
        DS -->|scrape| SS
        SS -->|raw text| RS["Raw Storage\nScraped Data"]:::storage
        SS -->|parsed PDF| RP["Raw Storage\nScraped_pdf_file"]:::storage
        RS -->|feeds| AN1["Initial Annotations"]:::annotation
        click SS "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/py_files_for_scraping/"
        click RS "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Scraped data/"
        click RP "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Scraped_pdf_file/"
    end

    subgraph "Test Data"
        direction TB
        Q["Gold Questions"]:::storage
        A["Reference Answers"]:::storage
        click Q "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/data/test/questions.txt"
        click A "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/data/test/reference_answers.txt"
    end

    DS --> SS
    Q --> AN1
    A --> AN1

    %% Annotation & Embedding
    subgraph "Annotation & Embedding"
        direction TB
        AN1 --> AN2["Final Annotations"]:::annotation
        click AN1 "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Annotations/"
        click AN2 "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Final_Annotations/"
        AN2 --> EMB["Embedding Generation"]:::annotation
        click EMB "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/Embeddings/chroma_embeddings_script.py"
        EMB --> VS["Vector Store\n(Chroma)"]:::storage
    end

    %% Model Execution
    subgraph "Model Execution"
        direction TB
        VS --> RP1["Retriever"]:::model
        RP1 --> RR["Reranker"]:::model
        RR --> GEN["LLM Generation\n(LLAMA2-70B-chat)"]:::model
        RP1 --> GEN
        CB["Closed-Book Baseline"]:::model
        click CB "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/RAG Pipeline/closed_book_model_script.py"
        CB --> PP
        click RP1 "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/RAG Pipeline/rag_pipeline.py"
        click RR "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/RAG Pipeline/rag_pipeline.py"
        click GEN "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/RAG Pipeline/rag_pipeline.py"
    end

    %% Post-Processing
    subgraph "Post-Processing"
        direction TB
        PP["Post-Processing"]:::model
        click PP "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/Postprocessing/postprocessing.py"
        GEN --> PP
    end

    %% Evaluation & Reporting
    subgraph "Evaluation & Reporting"
        direction TB
        PP --> EM["Evaluation Metrics"]:::eval
        PP --> ST["Significance Testing"]:::eval
        PP --> GA["Generated Answers"]:::eval
        PP --> SO["System Outputs"]:::eval
        PP --> RP["IAA Reporting"]:::eval

        click EM "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/evaluation_metrics/evaluation_metrics.py"
        click ST "https://github.com/akshay140601/end-to-end-nlp-system/blob/main/Significance Testing/significance_testing_script.py"
        click GA "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Generated answers/"
        click SO "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/system_outputs/"
        click RP "https://github.com/akshay140601/end-to-end-nlp-system/tree/main/Reports/"
    end

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#000;
    classDef storage fill:#B0C4DE,stroke:#000;
    classDef annotation fill:#90EE90,stroke:#000;
    classDef model fill:#FFA500,stroke:#000;
    classDef eval fill:#DDA0DD,stroke:#000;
    classDef external fill:#F0F0F0,stroke:#000;
```
