How larrysman/e-commerce-with-nlp works
Project Type and Purpose
• This is a self-contained, local “smart search” CLI tool for an e-commerce product catalog.
• It leverages NLP (TF-IDF + cosine similarity) to extract constraints and rank products.
• Purpose: Demonstrate modular NLP pipelines, easy to extend or swap in production components (vector DB, embeddings, UI).
File Structure
Top-level:
• README.md, requirements.txt, project_structure.py
• e-commerce-with-nlp/
Under e-commerce-with-nlp/:
• data/
– product.py (data model or sample catalog)
• src/
– main_product_search.py (CLI entry point)
– product_loader.py (loads JSON catalog into DataFrame)
– searchable_text.py (builds “search_text” per product)
– user_query_extraction.py (parses price/rating/category constraints)
– product_filter_by_extracted_user_query.py (applies filters)
– product_pipeline.py (orchestrates full pipeline: load → transform → rank)
– __init__.py
– experiment.ipynb (proof-of-concept notebook)
Extract Key Architecture Insights from README
• Separation of concerns: each module does one job (loader, extractor, filter, ranker).
• In-memory TF-IDF vectorizer + cosine similarity.
• Interactive CLI interface.
• Future extensibility points: vector DB, REST/UI layer.
Main Components
A. CLI Interface
– Component: main_product_search.py
– Receives raw user query, displays results.
B. Query Extraction
– Component: user_query_extraction
– Outputs price/rating/category constraints + free-text.
C. Data Ingestion
– Component: product_loader
– Loads PRODUCT_CATALOG into an in-memory DataFrame.
D. Searchable Text Builder
– Component: searchable_text
– Combines name/description/category/price/rating into a corpus string.
E. TF-IDF Vectorizer
– Component: product_pipeline (initializes vectorizer, builds TFIDF_matrix)
– Stores: vocabulary, document vectors.
F. Filtering
– Component: product_filter_by_extracted_user_query
– Takes DataFrame + constraints, outputs filtered subset.
G. Ranking
– Component: product_pipeline (or separate ranking module)
– Computes cosine similarity between query vector and product vectors.
H. Presentation
– CLI prints top-N products with metadata and scores.
Relationships & Data Flows
1. User types query → CLI
2. CLI → user_query_extraction → parsed constraints + text fragment
3. product_loader → DataFrame of all products
4. searchable_text → DataFrame enriched with “search_text” column
5. product_pipeline (startup) → TF-IDF vectorizer → TFIDF_matrix
6. CLI passes query text → TF-IDF vectorizer → query_vector
7. Filter module applies constraints → filtered DataFrame
8. Ranking module computes similarities → sorted results
9. CLI renders output
Architectural Patterns & Principles
• Pipeline pattern: clearly defined stages (load → transform → filter → rank → output)
• Separation of concerns: each Python module is single responsibility
• Dependency injection: pass vectorizer and DataFrame into ranking
• Extensibility points: swap TF-IDF for embeddings, add UI layer
Technologies & Frameworks
• Core: Python, pandas, scikit-learn (TfidfVectorizer), NumPy
• CLI: built-in argparse or input() loop
• Data model: simple JSON loaded via pandas
• Future: Streamlit/Flask + React, FAISS/Milvus