How jatin-mehra119/wasserstoff-aiinterntask works
Project Type and Purpose
• This is a full-stack Retrieval-Augmented Generation (RAG) chat application.
• Purpose: Ingest multi-format documents (PDF, text, images), build a FAISS vector store, perform semantic search + AI-powered chat with citations and theme analysis, and serve via a FastAPI backend + HTML/JS frontend (plus an MVP Streamlit UI).
File Structure
• Top-level
• rag_elements/ – Core RAG engine (enhanced_vectordb.py, config.py)
• backend/ – FastAPI service (entry point, Pydantic models, utils, route modules, runtime vector_store/)
• frontend/ – Static web assets (HTML, JS, CSS)
• streamlit_rag_app.py – MVP Streamlit UI
• requirements.txt, Dockerfile, CI workflows, tests/
• Patterns & Config
• Modular “routes” in backend indicating clear API layering
• vector_store/ folder shows disk-backed FAISS index + metadata
• enhanced_vectordb.py implements ingestion → chunking → embedding → FAISS index → search → AI integration
Core Components & Interactions
a. Client Layer (Browser)
• index.html, script.js, style.css
• Interacts via REST API (XHR/fetch) to FastAPI
b. Backend Layer (FastAPI)
• main.py: app startup, CORS, mounting routes
• models.py / utils.py: data schemas + helpers
• routes/:
• main_routes.py – serve static frontend + health check
• upload_routes.py – file-upload & directory processing
• chat_routes.py – chat endpoint, calls RAG engine
• store_routes.py – save/load FAISS vector store
c. RAG Engine Layer (rag_elements)
• EnhancedDocumentProcessor: process_documents(), create_vector_store(), search_documents(), get_chat_response(), save/load store
• Connects to FAISS library, SentenceTransformers, LangChain, GROQ Vision API for OCR & chat
d. Data Persistence
• FAISS index files (index.faiss, index.pkl) + enhanced_metadata.json in backend/vector_store/
e. External Services
• GROQ Vision API (OCR + LLM chat)
• Sentence Transformer models (embeddings)
• FAISS (local similarity search)
Architectural Patterns & Design Principles
• Layered Separation: UI ↔ API ↔ Core Engine
• Modular Routes: clear REST API surface
• Async I/O: FastAPI async endpoints + aiofiles
• Pluggable Engine: rag_elements easily reusable/testable
• Persistence decoupled: disk-backed vector store with load/save endpoints
• Single-page App style: frontend fetches JSON, renders chat & stats