How mildsauce9/documentor works
Project type and purpose
• DocuMentor is a full-stack web application that provides an intelligent, Retrieval-Augmented Generation (RAG) pipeline for natural-language Q&A over uploaded PDF documents.
• It consists of a frontend UI (static HTML/JS/CSS), a Python-based backend (app.py), local session persistence (file system), and calls out to the OpenAI API for embeddings and completions.
File structure
• Top-level files
• Dockerfile, .dockerignore – containerization
• requirements.txt – Python dependencies (likely includes streamlit or Flask, faiss-cpu, OpenAI SDK, PDF parser)
• app.py – main backend entrypoint
• README.md – project overview and setup
• static/
• index.html, app.js, styles.css – frontend assets served to the browser
• user_sessions/
• {session_uuid}/chat_history.json – persisted conversation turns
• {session_uuid}/faiss_index/index.faiss & index.pkl – Faiss vector store on disk
README for additional insights
• Describes RAG features: PDF upload, contextual memory, source-aware answers
• Instructs to run “python app.py” and browse to localhost:8501 (Streamlit’s default port)
• Requires OPENAI_API_KEY in .env for embeddings and completion calls
Main components
a. User (Web Browser)
b. Frontend UI
– Serves static files (HTML, JS, CSS)
– Handles file upload & chat interface
c. Backend Web Server (app.py inside Docker container)
• PDF Processor
– Accepts PDF → text extraction & chunking
• Embedding Service
– Calls OpenAI Embeddings API for each chunk
• Vector Store Manager
– Creates Faiss index, saves to user_sessions/{id}/faiss_index/
– Retrieves top-k vectors at query time
• RAG QA Engine
– Composes prompt with retrieved chunks + user question
– Calls OpenAI ChatCompletions API → generates answer
• Session Manager
– Generates session UUIDs
– Persists chat_history.json
d. Local Storage
– user_sessions directory on host
– Persists vector index files and chat history per session
e. External Service
– OpenAI API (Embeddings & ChatCompletions)
f. Containerization
– Dockerfile builds image, injecting ENV var OPENAI_API_KEY
Relationships and data flows
1) User uploads PDF via Frontend → HTTP POST to Backend
2) Backend extracts text, chunks, calls Embedding Service → stores vectors in Local Storage
3) Frontend sends user question → Backend Session Manager loads chat_history + Faiss index
4) RAG QA Engine retrieves relevant chunks → calls OpenAI Completion → returns answer
5) Backend appends question/answer to chat_history.json → Frontend displays with source references
Architectural patterns & design principles
• RAG (Retrieval-Augmented Generation) pipeline
• Single-page frontend & RESTful or WebSocket backend
• Per-session isolation via directory structure
• Layered separation: Presentation (UI), Application Logic (app.py), Persistence (file system), External API
• User / Browser (e.g., blue)
• Frontend components (e.g., green)
• Backend services (e.g., orange)
• Upload flow (Browser → Backend)
• Embedding & completion calls (Backend → OpenAI)
• Persistence interactions (Backend ↔ Local FS)
• Response flow (Backend → Browser)
• Protocols (HTTP, file I/O, REST API calls)