How akshaykumarbedre/chatwithanywebsite works
Project Type and Purpose
• This is a full-stack application designed for URL extraction/classification, description and product management, and interactive chatbot functionality. It uses AI-powered services to automate operations, helping businesses streamline content and product management while enhancing customer support.
Main Components
a. Frontend:
• Built with Next.js (as seen from package.json, next.config.mjs, tailwind, and component structure). It includes various pages (chatbot, admin, order placement, description, products, text-processor, url-processor, and an iframe variant) and shared UI components (Header, Footer).
b. Backend:
• A Flask-based server (app.py) that powers the API endpoints.
• Services/submodules that include:
• llm_service (for handling language model interactions),
• main_service (for core business logic),
• vector_store (handling vector databases with FAISS indices for description and product information),
• web_scraper (to extract and classify URLs).
• Streamlit application support (streamlit_app.py) indicating an alternative or additional UI for data visualization or quick demos.
c. Data Components:
• Raw, processed, and database folders under backend/data.
• Vector store indices are persisted in multiple places (inside backend/vectors, and within notebooks) using FAISS and Pickle files.
d. Configuration/Environment:
• Multiple config files like .env, configuration YAML, and Python configuration in src/config.
• Dependency files (requirements.txt, requirements_streamlit.txt, package.json) indicating the essential libraries.
e. Testing and Experimentation:
• Notebooks for experimentation in backend/notebooks, including specific examples for creating vector indices.
• Tests directory (backend/tests) with sample tests to verify functionality.
Relationships and Data Flow
• Users interact with the system via the Next.js frontend; their actions trigger API calls.
• The frontend makes calls (using utilities like the API file in frontend/src/utils/api.js) to the backend Flask server.
• The Flask backend receives requests, and based on the endpoint’s purpose, delegates tasks to the appropriate service:
• Web scraping service extracts URLs.
• Main service orchestrates operations like classification and CRUD operations for descriptions/products.
• Vector store service processes and retrieves vector embeddings stored as FAISS indices.
• LLM service generates context-aware responses for the chatbot.
• There’s also support via a Streamlit-based UI for interactions or monitoring.
• Data storage occurs via the vector indices and raw/processed data directories, allowing efficient retrieval for features like “View All Descriptions” or “View All Products.”
Architectural Patterns/Design Principles
• Separation of concerns: Frontend and backend are strictly separated (with dedicated folders).
• Modular service architecture within the backend: Each core feature (web scraping, vector storage, language model interactions) is isolated into its own service module.
• Use of external libraries and vector databases (FAISS) for efficient high-dimensional searches.
• Configuration and Environment management ensuring deployment consistency.