How tmtien31/history_question_generator works
Project type and purpose
• This is a full-stack web application (“History Question Generator”) built with FastAPI (backend) and Jinja2/HTML (frontend).
• Its purpose is to ingest Vietnamese history PDF documents, run a Retrieval-Augmented Generation (RAG) pipeline (LangChain + FAISS + Google Gemini 2.0), and produce multiple-choice question/answer CSVs for educational use.
File structure
• Top-level files:
• app.py: FastAPI server with upload and analysis endpoints
• requirements.txt, Dockerfile, .env → configuration & deployment
• README.md, LICENSE → documentation
• Directories:
• src/ (helper.py, prompt.py): core pipeline logic, prompt templates
• templates/index.html: Jinja2 frontend
• static/docs/: uploaded PDFs
• static/output/: generated CSVs
• data/: sample PDFs for testing
• research/: experimentation notebook
• Build & deploy artifacts: Dockerfile, .dockerignore, .gitignore
Extract architectural insights from README
• Architecture:
• Document upload → static/docs
• Text extraction (PyPDFLoader/PyPDF2)
• Chunking (TokenTextSplitter)
• Embeddings (HuggingFace) → FAISS vector store
• Generation (LangChain refine chain + Google Gemini 2.0)
• RetrievalQA chain for answers
• Output saved to CSV in static/output
• External dependencies: Google Gemini API, sentence-transformers, FAISS, LangChain
Main components
a. Client (Web Browser)
b. FastAPI Server (app.py + Uvicorn)
c. File storage
• PDF Store: static/docs
• CSV Store: static/output
d. Processing Pipeline
1. PDF Loader (PyPDFLoader/PyPDF2)
2. Text Chunking (TokenTextSplitter)
3. Embedding Generator (HuggingFace models)
4. Vector Database (FAISS)
5. RAG chains (LangChain refine & RetrievalQA)
e. External AI Service
• Google Gemini 2.0 API
f. Data Models & Helpers
• src/helper.py, src/prompt.py
Relationships and data flows
• Browser → HTTP POST /upload → FastAPI
• FastAPI writes PDF → static/docs
• FastAPI triggers processing:
• PDF Loader reads file → raw text
• TokenTextSplitter → text chunks
• HuggingFace embedding call → embedding vectors
• FAISS ingest & similarity search
• LangChain refine chain calls Google Gemini API → questions
• RetrievalQA chain uses FAISS & Gemini → answers
• FastAPI writes CSV → static/output
• Browser polls GET or page reload to download CSV
Architectural patterns & design principles
• Layered architecture: Presentation, API, Processing pipeline, Storage, External services
• Single-process monolith (all pipeline code in one service)
• RAG micro-pattern: separation of embedding store (FAISS) and LLM calls
• Asynchronous file I/O (aiofiles) and non-blocking endpoints
Technologies / frameworks to annotate
• FastAPI + Uvicorn (web framework & ASGI server)
• Jinja2 (templating)
• aiofiles (async file handling)
• PyPDFLoader, PyPDF2 (PDF extraction)
• TokenTextSplitter (LangChain)
• sentence-transformers (embeddings)
• FAISS (vector store)
• LangChain (refine & RetrievalQA chains)
• Google Gemini 2.0 Flash Lite API