# ussa24/document-based-qa-qg architecture

> Architecture diagram of the GitHub repository ussa24/document-based-qa-qg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ussa24/document-based-qa-qg
- Repository: https://github.com/ussa24/document-based-qa-qg

## Overview

##### Project type and purpose
- This is a backend-centric, full-stack-style AI service (no dedicated “frontend” UI here) that exposes a FastAPI HTTP interface.
- Purpose: ingest PDF documents, build a Retrieval-Augmented Generation (RAG) pipeline using LangChain + FAISS + LLaMA-2, generate QA pairs, evaluate answers, and expose results via REST endpoints.

##### File structure
- Top-level Python modules:
- preprocessing.py – PDF text extraction, chunking, embedding generation
- llm.py – orchestration of LLM calls for question generation and answering
- scoring.py – post-hoc answer evaluation algorithm (cosine-filter + voting with prompts)
- create_score_report.py – aggregates scores and generates PDF reports
- generators.py – helper routines for building/serializing Q&A data
- fast_api.py – FastAPI application with endpoints
- Supporting folders:
- uploads/ – incoming PDF documents by category
- score_reports/ – generated PDF reports for marketing, etc.
- images/ – static docs for README
- Config and dependencies: requirements.txt, README.md

##### README for architecture insights
- FastAPI endpoints store text in MongoDB, retrieve PDFs & categories, serve QA, trigger evaluation.
- External systems: MongoDB, FAISS (local), HuggingFace Models, google-trans.

##### Essential architectural elements in detail
- REST endpoints exposed by FastAPI on its boundary

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client
    Client["Client / User"]:::external

    %% FastAPI Service
    subgraph "FastAPI Service"
        API["FastAPI Service\n(endpoints:\nPOST /api/storetext\nGET /pdfs/{category}\nGET /categories\nGET /api/getquestions/{category}\nGET /validate_all)\n(fast_api.py)"]:::compute
    end

    %% Compute Modules
    subgraph "Compute Modules"
        Preproc["Preprocessing Module\n(preprocessing.py)"]:::compute
        LLM["LLM Module\n(llm.py)"]:::compute
        QA["Q&A Generators Module\n(generators.py)"]:::compute
        Scoring["Scoring Module\n(scoring.py)"]:::compute
        ReportGen["Score Report Generator\n(create_score_report.py)"]:::compute
    end

    %% Storage
    subgraph "Storage"
        Mongo[(MongoDB)]:::storage
        FAISS[(FAISS Vector Store)]:::storage
        Uploads["Raw PDF Storage\n(uploads/)"]:::storage
        ReportsFS["Generated PDF Reports\n(score_reports/)"]:::storage
    end

    %% External Services
    subgraph "External Services"
        HF["HuggingFace Models\n(LLaMA-2)"]:::external
        GT["google-trans"]:::external
    end

    %% Data Flows
    Client -->|calls REST| API
    API -->|invoke| Preproc
    API -->|invoke| LLM
    API -->|invoke| Scoring

    Preproc -->|extract & chunk| Uploads
    Preproc -->|store text metadata| Mongo
    Preproc -->|write embeddings| FAISS

    LLM -->|retrieve chunks| FAISS
    LLM -->|generate Q&A| HF
    LLM -->|post-process translations| GT
    LLM -->|store Q&A| Mongo
    QA -->|assist generation| LLM

    Scoring -->|read Q&A & answers| Mongo
    Scoring -->|compute scores| HF
    Scoring -->|aggregate results| Scoring
    Scoring -->|trigger report| ReportGen

    ReportGen -->|write PDF| ReportsFS
    ReportGen -->|read scores| Mongo

    %% Click Events
    click API "https://github.com/ussa24/document-based-qa-qg/blob/main/fast_api.py"
    click Preproc "https://github.com/ussa24/document-based-qa-qg/blob/main/preprocessing.py"
    click LLM "https://github.com/ussa24/document-based-qa-qg/blob/main/llm.py"
    click QA "https://github.com/ussa24/document-based-qa-qg/blob/main/generators.py"
    click Scoring "https://github.com/ussa24/document-based-qa-qg/blob/main/scoring.py"
    click ReportGen "https://github.com/ussa24/document-based-qa-qg/blob/main/create_score_report.py"
    click Uploads "https://github.com/ussa24/document-based-qa-qg/tree/main/uploads/"
    click ReportsFS "https://github.com/ussa24/document-based-qa-qg/tree/main/score_reports/"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
```
