How abhishekchauhan15/docuchat works
Main Components
• Flask API Server – Represent this as the front-facing component that accepts document uploads and query requests.
• Redis – Include this component as the central message broker and backend store that supports Celery task management.
• Weaviate – Illustrate this component as the vector database responsible for storing document embeddings and performing similarity searches.
• Ollama – Represent this as the component that generates embeddings from documents on demand.
• Supporting Infrastructure – Optionally, include Docker (via the Dockerfile and docker-compose.yml), configuration management (.env and config.py), and utility scripts (tasks.py, worker.py, utils.py) to show deployment and orchestration details.
Component Relationships and Interactions
• A user uploads a document via an HTTP POST request to the Flask API.
• The Flask API forwards the document task to Celery Workers using Redis as a broker.
• Asynchronous Processing:
• The Celery Workers receive the task, which then interacts with Ollama to generate embeddings.
• Once the embeddings are generated, they are stored in Weaviate.
• Query Process Flow:
• When a query is submitted, the Flask API communicates with Weaviate to retrieve the most relevant text chunks based on semantic similarity.
• Results are then returned via the Flask API back to the client.
• Additionally, track processing status (viewable by a task ID) and manage task status updates through Redis.
• Asynchronous Processing Pattern – Emphasize how Celery decouples long-running tasks from the synchronous request/response cycle, improving throughput and responsiveness.
• Microservices/Modular Components – Although contained within a single repository, the use of multiple specialized services (API, workers, vector database, LLM) reflects a modular approach.
Technology and Frameworks
• Flask (Python web framework) for the API.
• Celery (Python library) for background job processing.
• Redis for messaging and caching.
• Weaviate for vector search and storage.
• Ollama for local LLM-based embedding generation.
• Docker for containerization.