How aliaftab0786/langchain-chatbot works
Main Components and Their Roles
• Frontend: The Streamlit-powered UI found in the frontend directory (app.py) that serves as the user interface where queries are entered and responses are displayed.
• Chatbot Backend: The core logic of the RAG chatbot contained in chatbot/rag_chatbot.py. This component interacts with LangChain, handles API interactions with Anthropic (configured in chatbot/.env) and orchestrates retrieval augmented generation.
• Web Scraper: Located in the scraper directory, the Scrapy spiders (ecommerce_spider.py, quotes_spider.py) and related modules gather product data (information such as title, price, rating) from various e-commerce and content sources.
• Data Storage & Retrieval: The data folder includes persistent storage of ChromaDB data (binary and metadata files) and a SQLite database (chroma.sqlite3). Also, products.json serves as a structured file for product data. These are used to support the retrieval functionality in the chatbot’s process.
• External Services: Integration with Anthropic’s API for language generation and sites/sources that are scraped.
Relationships and Data Flow
• The Frontend sends user queries (such as "What’s the price of Storage Carry Case?") to the Chatbot Backend.
• The Chatbot Backend processes the query using a Retrieval-Augmented Generation approach. It accesses the persistent ChromaDB/SQLite storage to retrieve relevant product or scraped data.
• The backend might in real-time or periodically rely on data fed by the Scraper, which collects updates from various e-commerce sites. Data is then stored in the data layer.
• External API interactions occur when the backend needs to leverage Anthropic’s Claude (as configured via the .env file).
• Data flows are bidirectional between the Frontend and Backend as well as between the Backend and Data Storage. The Scraper independently feeds data into the storage, which the Chatbot Backend accesses.
Architectural Patterns and Design Principles
• Modular Design: Each component (frontend, chatbot logic, scraper, data storage) has its own directory, reflecting separation of concerns.
• Service-Oriented Data Flow: The front-end interacts with the backend via API calls (or script execution in the case of Streamlit), and the backend acts as an orchestrator between data retrieval (from storage) and external language generation.
• Data Persistence: Use of persistent storage (ChromaDB and SQLite) ensures data consistency and enables retrieval augmentation.
• Integration Pattern: Scraped data is used as a knowledge source for answering queries, integrating web scraping and chatbot processing in a retrieval-augmented manner.