How arpanbasusachdeva/querydocs works
Project Type and Purpose
• Project Type: Full-stack web application with AI/ML components
• Purpose: Provide a user interface (Streamlit) for uploading documents, storing them, generating embeddings, and enabling context-aware AI chat.
File Structure
Top-Level:
• main.py → FastAPI backend entry point
• streamlit_app.py → Streamlit frontend entry point
• Router/ → Core business logic modules
• database.py → Database connection setup
• table_creater.py → ORM models (SQLAlchemy)
• relations.py → Model relationships
• Chatbot_retriver.py → Chat logic (query embeddings, call AI)
• embedding.py → Embedding generation pipeline
• exception_utils.py→ Error handling utilities
• embedding/ → Persisted FAISS indexes and pickles
• uploads/ → File storage for uploaded documents
• requirements.txt → Python dependencies
• runtime.txt → Deployment runtime
Extract Insights from the README
• Backend: FastAPI + Uvicorn + SQLAlchemy + Pydantic
• Frontend: Streamlit + Pandas + Requests
• DB: SQLite (default) / PostgreSQL
• AI: Document embedding + semantic search + external LLM API
• Key features: Document management, context-aware chat, session/chat history, exception logging
Main Components
a. User Interface Layer
• Streamlit Frontend
b. API Layer
• FastAPI application (main.py) hosted on Uvicorn
• Pydantic schemas, routers
c. Persistence Layer
• Relational DB (SQLite/PostgreSQL)
• ORM models (SQLAlchemy)
• File storage (uploads directory)
• FAISS embedding indexes on disk
d. AI/ML Layer
• Embedding generation module (Router/embedding.py)
• FAISS index lookup (Router/Chatbot_retriver.py)
• External LLM API calls (e.g., OpenAI)
e. Utility/Support
• Exception handling (Router/exception_utils.py)
• Background tasks (if any)
Relationships & Data Flows
1. User interacts via Streamlit UI
2. Frontend issues HTTP requests (upload, chat) → FastAPI endpoints
3. Upload flow:
• API saves file to uploads/
• API invokes embedding pipeline
• Documents are chunked & embeddings generated
• Embeddings persisted as FAISS indexes + metadata stored in DB
4. Chat flow:
• API retrieves relevant FAISS index → semantic search
• Combines retrieved context + user query → calls LLM
• Stores chat history in DB
• Returns AI response to frontend
5. Diagnostics & Exceptions:
• All errors funnel through exception_utils → logs in DB
Architectural Patterns & Design Principles
• Layered Architecture: UI → API → Persistence/AI
• Modularization: “Router” folder separates concerns
• Repository/ORM pattern for DB access
• FAISS as specialized datastore for vector search
• Stateless HTTP services + session-based chat tracking
Technologies & Frameworks to Annotate
• Frontend: Streamlit, Pandas, Requests
• API: FastAPI, Uvicorn, Pydantic
• ORM: SQLAlchemy
• DB: SQLite / PostgreSQL
• Embedding Store: FAISS indexes (.faiss, .pkl files)
• External AI: OpenAI (or other LLM provider)