How saheba-freeji/simran-r-wasserstoff-aiinterntask works
Project Type and Purpose
• This is a full-stack “Document Theme Identifier Chatbot.”
• Frontend: Streamlit UI for uploading documents and chatting.
• Backend: FastAPI service that ingests documents, runs OCR, builds a vector store, queries an LLM, and identifies themes.
• Data: FAISS vector store for embeddings; local file store for original docs.
• External Services: Tesseract OCR and OpenAI GPT API.
File Structure
• Top‐level folders:
• backend/ – FastAPI code, models (api.py, document.py), service modules (document_parser.py, theme_identifier.py, vector_store.py, etc.), Dockerfile, data/vectorstore, files/
• frontend/ – Streamlit app (app.py), Dockerfile
• docker-compose.yml, start.sh – orchestrates frontend, backend, and any side‐cars
• Key patterns:
• Modular “services” in backend – single FastAPI process coordinating parsing, vector store, theme logic
• Separate concerns: config, models, services, data, files
README for Additional Insights
• Confirms architecture: FastAPI + Streamlit, FAISS, Tesseract, OpenAI
• Lists API endpoints (/upload, /chat, /documents) and UI flow tabs
• Describes containerized deployment via Docker Compose
Main Components & Their Relationships
a. Frontend (Streamlit)
• UI for file upload, query input, and displaying citations/themes
• Communicates via HTTP to Backend API endpoints
b. Backend (FastAPI)
• /upload → DocumentParser service:
• Reads DOCX, PDF, images
• Runs Tesseract OCR on scans
• Normalizes text, calls VectorStore service to index embeddings in FAISS
• /chat → ThemeIdentifier service:
• Receives user query
• Queries FAISS for relevant document chunks
• Calls OpenAI GPT API to extract themes and generate citations
• /documents → Lists available files and metadata
c. Data Storage
• FAISS vectorstore (faiss.index + metadata.pkl) for nearest‐neighbor embedding retrieval
• Local “files/” folder to hold original docs
d. External Services
• Tesseract OCR engine (local container or binary)
• OpenAI GPT (REST API)
Architectural Patterns & Design Principles
• Layered architecture:
• Presentation (Streamlit)
• API (FastAPI controllers)
• Business logic (document_parser, theme_identifier, vector_store services)
• Infrastructure (Vector store, file storage, external OCR/LLM)
• Separation of concerns: parsing vs. indexing vs. theme extraction
• Containerized micro‐deploy units orchestrated by Docker Compose
Elements to Include for Maximum Clarity
• Component labels with filenames or modules (e.g., DocumentParser service, ThemeIdentifier service)