# reverseengineeringdude/exampreparator architecture

> Architecture diagram of the GitHub repository reverseengineeringdude/exampreparator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reverseengineeringdude/exampreparator
- Repository: https://github.com/reverseengineeringdude/exampreparator

## Overview

##### Project type & purpose (what you’re diagramming)
This is a **single-service Flask web application** that provides an **exam-prep workflow**:

- Users upload **PDF notes** and **previous question papers**
- Backend performs **text extraction** (PyPDF2)
- Performs **question extraction + grouping** using **SBERT embeddings**
- Builds a **retrieval index** (FAISS) over uploaded notes
- Uses **RAG** to generate answers via **Gemini 1.5 Pro (API)** or a **local LLaMA 3** alternative
- Enforces **answer length/format constraints** based on marks (1/3/7)
- Persists outputs as **JSON artifacts** and exposes progress/quota warnings to the UI

##### File structure analysis → components to reflect
From the tree:

##### Backend boundary
- `backend/app.py`
Likely Flask routes: upload endpoints, status/progress endpoints, render templates, and trigger processing pipeline.
- `backend/services.py`
Core “service layer”: PDF parsing, embeddings, FAISS indexing, question clustering, batching, answer formatting, RAG calls, quota logic, progress updates.
- Artifact/state files in `backend/`:
- `study_material.json` (extracted notes content / chunks)
- `generated_answers.json` (final answers output)
- `progress.json` (pipeline status for UI)
- `flask_log.txt` (server logs)
- `dummy_note.txt`, `dummy_question.txt` (fixtures/dev)
- `requirements.txt` indicates Python deps (Flask, PyPDF2, FAISS, sentence-transformers, Gemini client, etc.)
- `render.yaml` indicates deployment target (Render) → include runtime/deployment node.

##### Frontend boundary (server-rendered pages + static assets)
- `templates/*.html`: `index.html`, `landingpage.html`, `uplaod.html` (typo but it’s a page), `answers.html`
- `static/scripts.js`, `static/styles.css`
JS likely polls progress/quota endpoints and renders updates.

This suggests the UI is **not a separate SPA service**; it’s Flask-rendered templates with JS enhancements.

##### User Browser
- Pages: Landing / Upload / Answers
- JS: progress polling, quota warnings display
##### Flask Web App (backend/app.py)
- Responsibilities: routing, file upload handling, triggers pipeline, serves templates/static, returns JSON status/results
##### Processing Services (backend/services.py)
- **PDF Text Extractor (PyPDF2)**
- **Chunker / Preprocessor** (implied; needed for embeddings + FAISS)
- **Embeddings Generator (SBERT / sentence-transformers)**
- **Question Extractor + Grouper (SBERT similarity / clustering)**
- **FAISS Index Builder + Retriever**
- **RAG Answer Generator**
- Prompting + context assembly
- **Batching layer** (rate-limit avoidance)
- **Mark-based Answer Formatter**
- 1-mark = 1 word
- 3-marks = short paragraph
- 7-marks = detailed essay
- **Fallback Answerer (KTU 2019 scheme guidelines)** when retrieval is insufficient
- **Progress/Quota Monitor** (writes progress.json, flags warnings)
##### Local Storage / Artifacts (filesystem / JSON files)
- `study_material.json`, `generated_answers.json`, `progress.json`, logs
##### External LLM Provider (Gemini 1.5 Pro API)
- Network boundary + rate limits/quota.
##### Optional Local Model Runtime (LLaMA 3 local)

- Browser → Flask: HTTP (upload PDFs, request answers page, poll progress)
- Flask → Services: in-process calls (same runtime)
- Services → Disk: read/write JSON artifacts + logs
- Services → Gemini API: HTTPS calls (batched payloads)
- Services → Local LLaMA runtime: local inference (optional)
- Services → FAISS index: local in-memory + serialized (if persisted; if not clear, show as in-process component with optional persistence)

##### Architectural pattern callouts
- **Monolith with internal service layer** (Flask + services.py)
- **Pipeline architecture** (extraction → embeddings → indexing → retrieval → generation → formatting)
- **RAG** using FAISS + LLM
- **Rate-limit aware batching** as a first-class component
- **State via JSON files** (instead of DB)

##### Upload
- User uploads Notes PDF + Question Paper PDF via Browser → Flask `/upload` (or equivalent)
##### Text Extraction
- Notes PDF → PyPDF2 → raw text
- QP PDF → PyPDF2 → raw text
##### Preprocessing
- Clean text, split into chunks (for embeddings + retrieval)
##### Question Extraction & Grouping
- Extract candidate questions from QP text
- SBERT embeddings → similarity grouping / dedupe / “important questions” sets
##### Indexing (Study material)
- SBERT embeddings for notes chunks
- Build FAISS index (vector store)
- Persist to `study_material.json` (and optionally index files if present later)
##### Retrieval (per question)
- Embed question → FAISS search → top-k relevant chunks
##### Generation
- Construct prompt with retrieved context
- Batch multiple questions per request to avoid rate limits
- Call Gemini API (or local LLaMA)
##### Fallback path
- If retrieval insufficient: generate based on KTU 2019 scheme guidelines
##### Persist outputs + progress
- Write `generated_answers.json`
- Update `progress.json` continuously
##### UI consumption
- Browser polls progress endpoint
- Browser renders quota warnings + completion
- Answers page reads from generated answers

- **Progress updates emitted after each stage** (to `progress.json` then to UI)
- **Quota/rate-limit checks** influencing batching/backoff behavior

- Render Web Service runs Flask app
- Persistent disk/volume if configured (otherwise ephemeral filesystem risk—call out that JSON artifacts may not persist across deploys)

That combination captures both the system boundary and the “why it works” mechanics (RAG + batching + formatting + progress).

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Diagram #1: Container / Architecture (C4 L2-ish)
  %% =========================

  U["User Browser"]:::ui
  Pages["Server-rendered Pages<br/>(Landing / Upload / Index / Answers)"]:::ui
  JS["JS Enhancements<br/>(progress polling, quota warnings)"]:::ui
  CSS["CSS Styles"]:::ui

  subgraph "Render Deployment (runtime)"
    RenderWS["Render Web Service<br/>(Flask runtime)"]:::deploy
  end

  subgraph "Flask Monolith (Single Service)"
    FlaskApp["Flask Web App<br/>(routes, uploads, templates, JSON endpoints)"]:::backend

    subgraph "Processing Service Layer (in-process)"
      Svc["Core Services"]:::backend

      PDF["PDF Text Extractor<br/>(PyPDF2)"]:::ml
      Prep["Chunker / Preprocessor"]:::ml
      QExtract["Question Extractor + Grouper<br/>(SBERT similarity/clustering)"]:::ml
      Embed["Embeddings Generator<br/>(SBERT / sentence-transformers)"]:::ml
      Faiss["FAISS Index Builder + Retriever"]:::ml
      RAG["RAG Answer Generator<br/>(prompt+context assembly)"]:::ml
      Batch["Batching / Backoff Layer<br/>(rate-limit mitigation)"]:::ml
      Format["Mark-based Answer Formatter<br/>(1/3/7 marks constraints)"]:::ml
      Fallback["Fallback Answerer<br/>(KTU 2019 scheme guidelines)"]:::ml
      Monitor["Progress/Quota Monitor<br/>(updates + warnings)"]:::ml
    end

    subgraph "Local Storage (JSON artifacts + logs)"
      Disk[("Local Disk / Persistent Volume")]:::store
      Study[("study_material.json")]:::store
      Answers[("generated_answers.json")]:::store
      Progress[("progress.json")]:::store
      Logs[("flask_log.txt")]:::store
    end
  end

  Gemini["Gemini 1.5 Pro API<br/>(external, quota/rate limits)"]:::external
  LocalLLM["Optional Local LLaMA 3 Runtime"]:::external

  %% UI composition
  U -->|"HTTP GET (HTML)"| FlaskApp
  FlaskApp -->|"renders"| Pages
  FlaskApp -->|"serves"| JS
  FlaskApp -->|"serves"| CSS
  U -->|"loads"| Pages
  U -->|"loads"| JS
  U -->|"loads"| CSS

  %% Upload + polling
  U -->|"HTTP multipart/form-data (PDF uploads)"| FlaskApp
  U -->|"HTTP GET/JSON (poll progress + quota warnings)"| FlaskApp
  FlaskApp -->|"JSON status/results"| U

  %% Backend internal calls
  RenderWS -->|"runs"| FlaskApp
  FlaskApp -->|"in-process calls (text/chunks/questions)"| Svc

  Svc -->|"extract text"| PDF
  Svc -->|"clean/split chunks"| Prep
  Svc -->|"extract+group questions"| QExtract
  Svc -->|"embed notes/questions"| Embed
  Svc -->|"build/search vectors"| Faiss
  Svc -->|"assemble context + prompts"| RAG
  RAG -->|"batch + backoff"| Batch
  Batch -->|"HTTPS (batched prompts/questions)"| Gemini
  RAG -->|"local inference (optional)"| LocalLLM
  Svc -->|"enforce length/format"| Format
  Svc -->|"fallback when context insufficient"| Fallback
  Svc -->|"progress + quota updates"| Monitor

  %% Persistence
  Disk --- Study
  Disk --- Answers
  Disk --- Progress
  Disk --- Logs

  Svc -->|"file write/read (JSON artifacts)"| Study
  Svc -->|"file write/read (JSON outputs)"| Answers
  Monitor -->|"file write (stage progress)"| Progress
  FlaskApp -->|"file read (progress for UI polling)"| Progress
  FlaskApp -->|"file read (answers for Answers page)"| Answers
  FlaskApp -->|"file append (server logs)"| Logs

  %% Notes
  Note1["Notes:<br/>- Monolith with internal service layer<br/>- RAG = FAISS retrieval + LLM generation<br/>- Batched requests mitigate rate limits<br/>- Progress via progress.json polled by UI"]:::note
  Monitor -.->|"emits warnings"| Note1

  %% =========================
  %% Diagram #2: Processing Pipeline (vertical stages)
  %% =========================

  subgraph "Processing Pipeline (end-to-end)"
    P1["1) Upload PDFs<br/>(Notes + Question Paper)"]:::step
    P2["2) Text Extraction<br/>(PyPDF2)"]:::step
    P3["3) Preprocessing<br/>(clean + chunk)"]:::step
    P4["4) Question Extraction & Grouping<br/>(SBERT embeddings + similarity)"]:::step
    P5["5) Indexing Study Material<br/>(SBERT + FAISS)"]:::step
    P6["6) Retrieval per Question<br/>(embed question → FAISS top-k)"]:::step
    P7["7) Generation (RAG)<br/>(context + prompt → LLM)"]:::step
    P8["8) Mark-based Formatting<br/>(1/3/7 constraints)"]:::step
    P9["9) Fallback if insufficient context<br/>(KTU 2019 guidelines)"]:::step
    P10["10) Persist outputs + progress<br/>(answers.json + progress.json)"]:::step
    P11["11) UI consumption<br/>(poll progress + render answers)"]:::step

    P1 --> P2 --> P3 --> P4 --> P5 --> P6 --> P7 --> P8 --> P9 --> P10 --> P11
  end

  %% Progress reporting after each stage
  P2 -.->|"update progress"| Progress
  P3 -.->|"update progress"| Progress
  P4 -.->|"update progress"| Progress
  P5 -.->|"update progress"| Progress
  P6 -.->|"update progress"| Progress
  P7 -.->|"update progress + quota checks"| Progress
  P8 -.->|"update progress"| Progress
  P9 -.->|"update progress"| Progress
  P10 -->|"write"| Answers

  %% Connect pipeline to runtime components
  U -->|"triggers pipeline"| P1
  P2 -->|"uses"| PDF
  P3 -->|"uses"| Prep
  P4 -->|"uses"| QExtract
  P5 -->|"uses"| Embed
  P5 -->|"uses"| Faiss
  P7 -->|"calls (batched)"| Gemini
  P7 -.->|"optional"| LocalLLM
  P8 -->|"uses"| Format
  P9 -->|"uses"| Fallback
  P10 -->|"uses"| Monitor
  P11 -->|"polls"| FlaskApp

  %% =========================
  %% Diagram #3: Deployment / Repo Artifacts (supporting)
  %% =========================

  subgraph "Repo Config & Tests (non-runtime but important)"
    Req["Python Dependencies<br/>(requirements.txt)"]:::support
    RenderY["Render Deployment Config<br/>(render.yaml)"]:::support
    Tests["Tests<br/>(upload, marks, quota, errors)"]:::support
    Fixture["Question Paper Fixture<br/>(test input)"]:::support
    DummyN["Dummy Notes Fixture"]:::support
    DummyQ["Dummy Question Fixture"]:::support
    SamplePDF["Sample PDF Input"]:::support
  end

  Req -.->|"installs deps for"| RenderWS
  RenderY -.->|"defines service"| RenderWS
  Tests -.->|"verifies behavior of"| FlaskApp
  Fixture -.->|"input for"| Tests
  DummyN -.->|"dev input for"| FlaskApp
  DummyQ -.->|"dev input for"| FlaskApp
  SamplePDF -.->|"example upload"| FlaskApp

  %% =========================
  %% Click Events (from component_mapping)
  %% =========================
  click FlaskApp "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/app.py"
  click Svc "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/services.py"

  click Study "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/study_material.json"
  click Answers "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/generated_answers.json"
  click Progress "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/progress.json"
  click Logs "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/flask_log.txt"

  click DummyN "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/dummy_note.txt"
  click DummyQ "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/dummy_question.txt"
  click SamplePDF "https://github.com/reverseengineeringdude/exampreparator/blob/main/6131A (1).pdf"

  click Pages "https://github.com/reverseengineeringdude/exampreparator/blob/main/templates/index.html"
  click JS "https://github.com/reverseengineeringdude/exampreparator/blob/main/static/scripts.js"
  click CSS "https://github.com/reverseengineeringdude/exampreparator/blob/main/static/styles.css"

  click Req "https://github.com/reverseengineeringdude/exampreparator/blob/main/requirements.txt"
  click RenderY "https://github.com/reverseengineeringdude/exampreparator/blob/main/render.yaml"

  click Tests "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_api_upload.py"
  click Fixture "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_question_paper_input.txt"

  %% Additional click coverage for mapped template/test files
  click P11 "https://github.com/reverseengineeringdude/exampreparator/blob/main/templates/answers.html"
  click P1 "https://github.com/reverseengineeringdude/exampreparator/blob/main/templates/uplaod.html"
  click U "https://github.com/reverseengineeringdude/exampreparator/blob/main/templates/landingpage.html"

  click Format "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_marks.py"
  click Batch "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_quota.py"
  click Monitor "https://github.com/reverseengineeringdude/exampreparator/blob/main/backend/test_weighting.py"
  click PDF "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_direct.py"
  click Note1 "https://github.com/reverseengineeringdude/exampreparator/blob/main/test_err.py"

  %% =========================
  %% Styles (color-coded)
  %% =========================
  classDef ui fill:#1e88e5,stroke:#0d47a1,color:#ffffff,stroke-width:1.5px
  classDef backend fill:#2e7d32,stroke:#1b5e20,color:#ffffff,stroke-width:1.5px
  classDef ml fill:#6a1b9a,stroke:#4a148c,color:#ffffff,stroke-width:1.5px
  classDef store fill:#546e7a,stroke:#263238,color:#ffffff,stroke-width:1.5px
  classDef external fill:#ef6c00,stroke:#e65100,color:#ffffff,stroke-width:1.5px
  classDef deploy fill:#00897b,stroke:#004d40,color:#ffffff,stroke-width:1.5px
  classDef support fill:#5d4037,stroke:#3e2723,color:#ffffff,stroke-width:1.5px
  classDef step fill:#7b1fa2,stroke:#4a148c,color:#ffffff,stroke-width:1.2px
  classDef note fill:#f9a825,stroke:#f57f17,color:#1b1b1b,stroke-width:1.2px
```
