How reverseengineeringdude/exampreparator works
Project type & purpose (what you’re diagramming)
This is a single-service Flask web application that provides an exam-prep workflow:
• Users upload PDF notes and previous question papers
• Backend performs text extraction (PyPDF2)
• Performs question extraction + grouping using SBERT embeddings
• Builds a retrieval index (FAISS) over uploaded notes
• Uses RAG to generate answers via Gemini 1.5 Pro (API) or a local LLaMA 3 alternative
• Enforces answer length/format constraints based on marks (1/3/7)
• Persists outputs as JSON artifacts and exposes progress/quota warnings to the UI
File structure analysis → components to reflect
From the tree:
Backend boundary
• backend/app.py
Likely Flask routes: upload endpoints, status/progress endpoints, render templates, and trigger processing pipeline.
• backend/services.py
Core “service layer”: PDF parsing, embeddings, FAISS indexing, question clustering, batching, answer formatting, RAG calls, quota logic, progress updates.
• Artifact/state files in backend/:
• study_material.json (extracted notes content / chunks)
• generated_answers.json (final answers output)
• progress.json (pipeline status for UI)
• flask_log.txt (server logs)
• dummy_note.txt, dummy_question.txt (fixtures/dev)
• requirements.txt indicates Python deps (Flask, PyPDF2, FAISS, sentence-transformers, Gemini client, etc.)
• render.yaml indicates deployment target (Render) → include runtime/deployment node.
Frontend boundary (server-rendered pages + static assets)
• templates/*.html: index.html, landingpage.html, uplaod.html (typo but it’s a page), answers.html
• static/scripts.js, static/styles.css
JS likely polls progress/quota endpoints and renders updates.
This suggests the UI is not a separate SPA service; it’s Flask-rendered templates with JS enhancements.
User Browser
• Pages: Landing / Upload / Answers
• JS: progress polling, quota warnings display
Flask Web App (backend/app.py)
• Responsibilities: routing, file upload handling, triggers pipeline, serves templates/static, returns JSON status/results
Processing Services (backend/services.py)
• PDF Text Extractor (PyPDF2)
• Chunker / Preprocessor (implied; needed for embeddings + FAISS)
• Embeddings Generator (SBERT / sentence-transformers)
• Question Extractor + Grouper (SBERT similarity / clustering)
• FAISS Index Builder + Retriever
• RAG Answer Generator
• Prompting + context assembly
• Batching layer (rate-limit avoidance)
• Mark-based Answer Formatter
• 1-mark = 1 word
• 3-marks = short paragraph
• 7-marks = detailed essay
• Fallback Answerer (KTU 2019 scheme guidelines) when retrieval is insufficient
• Progress/Quota Monitor (writes progress.json, flags warnings)
Local Storage / Artifacts (filesystem / JSON files)
• study_material.json, generated_answers.json, progress.json, logs
External LLM Provider (Gemini 1.5 Pro API)
• Network boundary + rate limits/quota.
Optional Local Model Runtime (LLaMA 3 local)
• Browser → Flask: HTTP (upload PDFs, request answers page, poll progress)
• Flask → Services: in-process calls (same runtime)
• Services → Disk: read/write JSON artifacts + logs
• Services → Gemini API: HTTPS calls (batched payloads)
• Services → Local LLaMA runtime: local inference (optional)
• Services → FAISS index: local in-memory + serialized (if persisted; if not clear, show as in-process component with optional persistence)
Architectural pattern callouts
• Monolith with internal service layer (Flask + services.py)
• Pipeline architecture (extraction → embeddings → indexing → retrieval → generation → formatting)
• RAG using FAISS + LLM
• Rate-limit aware batching as a first-class component
• State via JSON files (instead of DB)
Upload
• User uploads Notes PDF + Question Paper PDF via Browser → Flask /upload (or equivalent)
Text Extraction
• Notes PDF → PyPDF2 → raw text
• QP PDF → PyPDF2 → raw text
Preprocessing
• Clean text, split into chunks (for embeddings + retrieval)
Question Extraction & Grouping
• Extract candidate questions from QP text
• SBERT embeddings → similarity grouping / dedupe / “important questions” sets
Indexing (Study material)
• SBERT embeddings for notes chunks
• Build FAISS index (vector store)
• Persist to study_material.json (and optionally index files if present later)
Retrieval (per question)
• Embed question → FAISS search → top-k relevant chunks
Generation
• Construct prompt with retrieved context
• Batch multiple questions per request to avoid rate limits
• Call Gemini API (or local LLaMA)
Fallback path
• If retrieval insufficient: generate based on KTU 2019 scheme guidelines
Persist outputs + progress
• Write generated_answers.json
• Update progress.json continuously
UI consumption
• Browser polls progress endpoint
• Browser renders quota warnings + completion
• Answers page reads from generated answers
• Progress updates emitted after each stage (to progress.json then to UI)
• Quota/rate-limit checks influencing batching/backoff behavior
• Render Web Service runs Flask app
• Persistent disk/volume if configured (otherwise ephemeral filesystem risk—call out that JSON artifacts may not persist across deploys)
That combination captures both the system boundary and the “why it works” mechanics (RAG + batching + formatting + progress).