How sahilnegi4444/ecomm-qa works
1. What this repo is
- ecomm-QA is an end-to-end retrieval-augmented generation (RAG) system for e-commerce product Q&A and recommendations.
- It is split into two major boundaries: an offline ingestion/indexing path and an online chat/query path.
2. Offline data source and preparation
- The data pipeline starts from a MySQL product catalog with ~50K products.
- The notebooks in data/cleaning1.ipynb, data/cleaning2.ipynb, and experiments.ipynb appear to be used for exploration, cleaning, and schema/data-shaping before production ingestion.
3. Ingestion pipeline
- src/ingest.py and src/pipeline/pipeline.py orchestrate the ETL flow from raw product records into searchable documents.
- src/processing/preprocessing.py is the data-cleaning boundary: it likely normalizes product text, aggregates review content, and prepares fields such as title, category, price, and description for embedding.
4. Embedding generation
- src/embeddings/embeddings.py is the embedding boundary.
- It uses sentence-transformer style embeddings, producing dense vectors for each product record so that product meaning can be searched semantically rather than by keyword.
5. Vector database layer
- src/db/vector_db.py is the persistence and retrieval store boundary.
- The README indicates ChromaDB is used as a persistent vector database, storing product embeddings plus metadata such as title, price, and category.
- This layer is the offline/online shared asset: ingestion writes to it, query-time retrieval reads from it.
6. Semantic retrieval
- src/retriever.py is the online retrieval component.
- It converts the user query into an embedding, performs cosine similarity search against ChromaDB, and returns the top-k most relevant products.
- This is the primary interface between user intent and the product corpus.
7. RAG orchestration and session memory
- src/rag_pipeline/chatbot.py is the central orchestrator for the conversational flow.
- It combines the current user query, prior chat context, and retrieved product candidates, and it also handles follow-up detection / conversation state so users can ask multi-turn questions like “what about cheaper options?”
8. LLM response generation
- src/llm/chat.py is the generation boundary.
- The README indicates the system uses Ollama with Llama 3.1 8B locally or via a model-serving runtime.
- The generator receives a prompt containing conversation history plus retrieved products and produces the final natural-language answer.
9. API and client-facing layer
- src/app/main.py is the FastAPI entrypoint.
- The API exposes the chat surface and a health check endpoint; the README explicitly calls out POST /chat and GET /health.
- src/app/static/index.html suggests a lightweight static frontend is served from the same application rather than a separate web app.
- FastAPI/Pydantic form the request validation and HTTP boundary around the RAG pipeline.
10. Operations, packaging, and test boundary
- deployment/docker/dockerfile packages the stack for reproducible deployment.
- src/logger.py centralizes application logging for pipeline/API visibility.
- src/test/test_chatbot.py provides a test boundary around chatbot behavior, likely focused on the orchestration layer rather than the whole vector pipeline.
- Overall, the repo is architected as a modular Python service with clear separation between data preparation, vector indexing, retrieval, LLM generation, and the FastAPI interface.
Main entry points: main.py Chat API · retriever.py Retriever · chatbot.py Chatbot · pipeline.py Pipeline
Components 18
Offline ingestion
Data preparation, embedding, and indexing into the vector store.
- Product catalog · Primary product data source used to build the Q&A index.
- Notebooks · Cleaning and experiment notebooks used to inspect and shape product data.
data/cleaning1.ipynb - Ingest entry · Top-level ingestion script that starts the offline indexing flow.
src/ingest.py - Pipeline · Coordinates the ingestion stages from raw records to indexed documents.
src/pipeline/pipeline.py - Preprocessing · Normalizes product fields and prepares records for embedding.
src/processing/preprocessing.py - Embeddings · Produces dense sentence-transformer style embeddings for products.
src/embeddings/embeddings.py - Vector DB · Persistent vector store for embeddings plus product metadata.
src/db/vector_db.py
Online serving
Chat API, retrieval, orchestration, and answer generation.
- Retriever · Embeds the query and retrieves top-k similar products from the vector store.
src/retriever.py - Chatbot · Combines user query, prior context, and retrieved candidates for multi-turn chat.
src/rag_pipeline/chatbot.py - Chat API · HTTP entrypoint exposing chat and health endpoints.
src/app/main.py - Static UI · Lightweight static page served by the application.
src/app/static/index.html - LLM chat · Generates the final natural-language response from prompt plus retrieved context.
src/llm/chat.py
Shared core
Modules used across ingestion and query-time flows.
- Logger · Central logging used across pipeline and API components.
src/logger.py - Package init · Repository package root marker.
src/__init__.py - App init · Application package marker.
src/app/__init__.py - DB init · Database package marker.
src/db/__init__.py
Operations
Packaging and validation boundaries.
- Test chatbot · Focused test coverage for chatbot orchestration behavior.
src/test/test_chatbot.py - Dockerfile · Container image definition for reproducible deployment.
deployment/docker/dockerfile
Connections 18
- Product catalog → Notebooks: explore/clean · Notebook work prepares raw catalog data for production ingestion.
- Notebooks → Ingest entry: shape input · Cleaned and structured product data feeds the ingestion entrypoint.
- Ingest entry → Pipeline: orchestrates · The entrypoint launches the ETL pipeline.
- Pipeline → Preprocessing: cleans · Pipeline invokes preprocessing to normalize product records.
- Preprocessing → Embeddings: prepares vectors · Cleaned product text is converted into embeddings.
- Embeddings → Vector DB: store vectors · Embeddings and metadata are written to the vector database.
- Vector DB → Retriever: serve retrieval · The vector store is queried during semantic search.
- Retriever → Chatbot: pass context · Retrieved products are handed to the chatbot orchestrator.
- Chatbot → LLM chat: prompt LLM · The orchestrator builds the prompt and calls the generator.
- Chat API → Chatbot: handle chat · API requests enter the conversational orchestrator.
- Chat API → Retriever: query products · The service triggers semantic retrieval for user questions.
- Chat API → Static UI: serve UI · The application serves a lightweight static frontend.
- Chat API → Logger: log activity · API and chat activity are recorded through the shared logger.
- Chatbot → Retriever: retrieve again · Multi-turn follow-ups can request another retrieval pass.
- Pipeline → Logger: log pipeline · Ingestion stages emit logs for visibility.
- Retriever → Logger: log search · Retrieval events are logged for debugging and tracing.
- Test chatbot → Chatbot: verify flow · Tests target the chatbot orchestration boundary.
- Dockerfile → Chat API: deploy app · Containerization packages the API and serving stack.