# sahilnegi4444/ecomm-qa architecture

> Architecture diagram of the GitHub repository sahilnegi4444/ecomm-qa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahilnegi4444/ecomm-qa
- Repository: https://github.com/sahilnegi4444/ecomm-qa

## Overview

1. **What this repo is**
   - `ecomm-QA` is an end-to-end retrieval-augmented generation (RAG) system for e-commerce product Q&A and recommendations.
   - It is split into two major boundaries: an **offline ingestion/indexing path** and an **online chat/query path**.

2. **Offline data source and preparation**
   - The data pipeline starts from a **MySQL product catalog** with ~50K products.
   - The notebooks in `data/cleaning1.ipynb`, `data/cleaning2.ipynb`, and `experiments.ipynb` appear to be used for exploration, cleaning, and schema/data-shaping before production ingestion.

3. **Ingestion pipeline**
   - `src/ingest.py` and `src/pipeline/pipeline.py` orchestrate the ETL flow from raw product records into searchable documents.
   - `src/processing/preprocessing.py` is the data-cleaning boundary: it likely normalizes product text, aggregates review content, and prepares fields such as title, category, price, and description for embedding.

4. **Embedding generation**
   - `src/embeddings/embeddings.py` is the embedding boundary.
   - It uses sentence-transformer style embeddings, producing dense vectors for each product record so that product meaning can be searched semantically rather than by keyword.

5. **Vector database layer**
   - `src/db/vector_db.py` is the persistence and retrieval store boundary.
   - The README indicates **ChromaDB** is used as a persistent vector database, storing product embeddings plus metadata such as title, price, and category.
   - This layer is the offline/online shared asset: ingestion writes to it, query-time retrieval reads from it.

6. **Semantic retrieval**
   - `src/retriever.py` is the online retrieval component.
   - It converts the user query into an embedding, performs cosine similarity search against ChromaDB, and returns the top-k most relevant products.
   - This is the primary interface between user intent and the product corpus.

7. **RAG orchestration and session memory**
   - `src/rag_pipeline/chatbot.py` is the central orchestrator for the conversational flow.
   - It combines the current user query, prior chat context, and retrieved product candidates, and it also handles follow-up detection / conversation state so users can ask multi-turn questions like “what about cheaper options?”

8. **LLM response generation**
   - `src/llm/chat.py` is the generation boundary.
   - The README indicates the system uses **Ollama** with **Llama 3.1 8B** locally or via a model-serving runtime.
   - The generator receives a prompt containing conversation history plus retrieved products and produces the final natural-language answer.

9. **API and client-facing layer**
   - `src/app/main.py` is the FastAPI entrypoint.
   - The API exposes the chat surface and a health check endpoint; the README explicitly calls out `POST /chat` and `GET /health`.
   - `src/app/static/index.html` suggests a lightweight static frontend is served from the same application rather than a separate web app.
   - FastAPI/Pydantic form the request validation and HTTP boundary around the RAG pipeline.

10. **Operations, packaging, and test boundary**
   - `deployment/docker/dockerfile` packages the stack for reproducible deployment.
   - `src/logger.py` centralizes application logging for pipeline/API visibility.
   - `src/test/test_chatbot.py` provides a test boundary around chatbot behavior, likely focused on the orchestration layer rather than the whole vector pipeline.
   - Overall, the repo is architected as a modular Python service with clear separation between data preparation, vector indexing, retrieval, LLM generation, and the FastAPI interface.

Main entry points: [`src/app/main.py`](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/main.py) (Chat API), [`src/retriever.py`](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/retriever.py) (Retriever), [`src/rag_pipeline/chatbot.py`](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/rag_pipeline/chatbot.py) (Chatbot), [`src/pipeline/pipeline.py`](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/pipeline/pipeline.py) (Pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_offline["Offline ingestion"]
  node_n1[("Product catalog<br/>mysql source")]
  node_n2["Notebooks<br/>exploration<br/>[cleaning1.ipynb]"]
  node_n3["Ingest entry<br/>etl entrypoint<br/>[ingest.py]"]
  node_n4["Pipeline<br/>etl orchestration<br/>[pipeline.py]"]
  node_n5["Preprocessing<br/>data cleaning<br/>[preprocessing.py]"]
  node_n6["Embeddings<br/>vectorization<br/>[embeddings.py]"]
  node_n7[("Vector DB<br/>chroma persistence<br/>[vector_db.py]")]
end

subgraph group_online["Online serving"]
  node_n8["Retriever<br/>semantic search<br/>[retriever.py]"]
  node_n9["Chatbot<br/>rag orchestrator<br/>[chatbot.py]"]
  node_n10["Chat API<br/>fastapi service<br/>[main.py]"]
  node_n11["Static UI<br/>frontend shell<br/>[index.html]"]
  node_n12["LLM chat<br/>ollama generation<br/>[chat.py]"]
end

subgraph group_shared["Shared core"]
  node_n13["Logger<br/>observability<br/>[logger.py]"]
  node_n14["Package init<br/>python package<br/>[__init__.py]"]
  node_n15["App init<br/>python package<br/>[__init__.py]"]
  node_n16["DB init<br/>python package<br/>[__init__.py]"]
end

subgraph group_ops["Operations"]
  node_n17["Test chatbot<br/>behavior test<br/>[test_chatbot.py]"]
  node_n18["Dockerfile<br/>deployment"]
end

node_n1 -->|"explore/clean"| node_n2
node_n2 -->|"shape input"| node_n3
node_n3 -->|"orchestrates"| node_n4
node_n4 -->|"cleans"| node_n5
node_n5 -->|"prepares vectors"| node_n6
node_n6 -->|"store vectors"| node_n7
node_n7 -->|"serve retrieval"| node_n8
node_n8 -->|"pass context"| node_n9
node_n9 -->|"prompt LLM"| node_n12
node_n10 -->|"handle chat"| node_n9
node_n10 -->|"query products"| node_n8
node_n10 -->|"serve UI"| node_n11
node_n10 -->|"log activity"| node_n13
node_n9 -->|"retrieve again"| node_n8
node_n4 -->|"log pipeline"| node_n13
node_n8 -->|"log search"| node_n13
node_n17 -->|"verify flow"| node_n9
node_n18 -->|"deploy app"| node_n10

click node_n2 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/data/cleaning1.ipynb"
click node_n3 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/ingest.py"
click node_n4 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/pipeline/pipeline.py"
click node_n5 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/processing/preprocessing.py"
click node_n6 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/embeddings/embeddings.py"
click node_n7 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/db/vector_db.py"
click node_n8 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/retriever.py"
click node_n9 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/rag_pipeline/chatbot.py"
click node_n10 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/main.py"
click node_n11 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/static/index.html"
click node_n12 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/llm/chat.py"
click node_n13 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/logger.py"
click node_n14 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/__init__.py"
click node_n15 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/__init__.py"
click node_n16 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/db/__init__.py"
click node_n17 "https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/test/test_chatbot.py"
click node_n18 "https://github.com/sahilnegi4444/ecomm-qa/tree/main/deployment/docker/dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5,node_n6,node_n7 toneBlue
class node_n8,node_n9,node_n10,node_n11,node_n12 toneAmber
class node_n13,node_n14,node_n15,node_n16 toneMint
class node_n17,node_n18 toneRose
```

## Components

### Offline ingestion

Data preparation, embedding, and indexing into the vector store.

- **Product catalog**: mysql source · Primary product data source used to build the Q&A index.
- [Notebooks](https://github.com/sahilnegi4444/ecomm-qa/blob/main/data/cleaning1.ipynb): exploration · Cleaning and experiment notebooks used to inspect and shape product data. · `data/cleaning1.ipynb`
- [Ingest entry](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/ingest.py): etl entrypoint · Top-level ingestion script that starts the offline indexing flow. · `src/ingest.py`
- [Pipeline](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/pipeline/pipeline.py): etl orchestration · Coordinates the ingestion stages from raw records to indexed documents. · `src/pipeline/pipeline.py`
- [Preprocessing](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/processing/preprocessing.py): data cleaning · Normalizes product fields and prepares records for embedding. · `src/processing/preprocessing.py`
- [Embeddings](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/embeddings/embeddings.py): vectorization · Produces dense sentence-transformer style embeddings for products. · `src/embeddings/embeddings.py`
- [Vector DB](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/db/vector_db.py): chroma persistence · Persistent vector store for embeddings plus product metadata. · `src/db/vector_db.py`

### Online serving

Chat API, retrieval, orchestration, and answer generation.

- [Retriever](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/retriever.py): semantic search · Embeds the query and retrieves top-k similar products from the vector store. · `src/retriever.py`
- [Chatbot](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/rag_pipeline/chatbot.py): rag orchestrator · Combines user query, prior context, and retrieved candidates for multi-turn chat. · `src/rag_pipeline/chatbot.py`
- [Chat API](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/main.py): fastapi service · HTTP entrypoint exposing chat and health endpoints. · `src/app/main.py`
- [Static UI](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/static/index.html): frontend shell · Lightweight static page served by the application. · `src/app/static/index.html`
- [LLM chat](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/llm/chat.py): ollama generation · Generates the final natural-language response from prompt plus retrieved context. · `src/llm/chat.py`

### Shared core

Modules used across ingestion and query-time flows.

- [Logger](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/logger.py): observability · Central logging used across pipeline and API components. · `src/logger.py`
- [Package init](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/__init__.py): python package · Repository package root marker. · `src/__init__.py`
- [App init](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/app/__init__.py): python package · Application package marker. · `src/app/__init__.py`
- [DB init](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/db/__init__.py): python package · Database package marker. · `src/db/__init__.py`

### Operations

Packaging and validation boundaries.

- [Test chatbot](https://github.com/sahilnegi4444/ecomm-qa/blob/main/src/test/test_chatbot.py): behavior test · Focused test coverage for chatbot orchestration behavior. · `src/test/test_chatbot.py`
- [Dockerfile](https://github.com/sahilnegi4444/ecomm-qa/tree/main/deployment/docker/dockerfile): deployment · Container image definition for reproducible deployment. · `deployment/docker/dockerfile`

## Connections

- Product catalog → Notebooks: explore/clean · Notebook work prepares raw catalog data for production ingestion.
- Notebooks → Ingest entry: shape input · Cleaned and structured product data feeds the ingestion entrypoint.
- Ingest entry → Pipeline: orchestrates · The entrypoint launches the ETL pipeline.
- Pipeline → Preprocessing: cleans · Pipeline invokes preprocessing to normalize product records.
- Preprocessing → Embeddings: prepares vectors · Cleaned product text is converted into embeddings.
- Embeddings → Vector DB: store vectors · Embeddings and metadata are written to the vector database.
- Vector DB → Retriever: serve retrieval · The vector store is queried during semantic search.
- Retriever → Chatbot: pass context · Retrieved products are handed to the chatbot orchestrator.
- Chatbot → LLM chat: prompt LLM · The orchestrator builds the prompt and calls the generator.
- Chat API → Chatbot: handle chat · API requests enter the conversational orchestrator.
- Chat API → Retriever: query products · The service triggers semantic retrieval for user questions.
- Chat API → Static UI: serve UI · The application serves a lightweight static frontend.
- Chat API → Logger: log activity · API and chat activity are recorded through the shared logger.
- Chatbot → Retriever: retrieve again · Multi-turn follow-ups can request another retrieval pass.
- Pipeline → Logger: log pipeline · Ingestion stages emit logs for visibility.
- Retriever → Logger: log search · Retrieval events are logged for debugging and tracing.
- Test chatbot → Chatbot: verify flow · Tests target the chatbot orchestration boundary.
- Dockerfile → Chat API: deploy app · Containerization packages the API and serving stack.
