How swagath18/genai-ecom-review-analyzer works
Main Components
• User Interface / Client: Although not a web UI per se, the “user” represents the data scientist or e-commerce professional triggering the pipeline (via CLI, script execution, or API call).
• Data Ingestion & Preprocessing: Includes files such as data_loader.py and Datacleaning.ipynb that load and clean the Walmart product review dataset (with the cleaned data saved into cleaneddata.csv).
• Core Analysis Pipeline:
• Review Analysis Module: review_analyzer.py which processes customer reviews using NLP techniques.
• NLP Modules: Leveraging advanced libraries like Transformers and Langchain, which are used for sentiment analysis and review summarization.
• ML Models: Embedded within the analysis pipeline where sentiment analysis, text summarization, and key metrics calculations take place.
• Deployment Component: deploy.py and main.py serve as the orchestration layer, setting up the runtime environment and starting the process.
• MLOps & Monitoring Module:
• MLflow Integration: Evidenced by mlops_tracker.py and the mlruns directory, this component logs performance metrics (e.g., total reviews, sentiment distribution, and average rating) and hyperparameters.
• Continuous Monitoring: Through MLflow’s tracking dashboard, where metrics (accuracy, loss, etc.) are recorded to ensure continuous oversight of model performance.
• External Dependencies: Libraries like MLflow, Langchain, and Transformers are core to the system’s operations.
Relationships & Data Flow
• Deployment → Data Ingestion & Preprocessing: Once initiated, the system ingests raw data (data_loader.py) and cleans it (Datacleaning.ipynb becomes the source for the cleaneddata.csv).
• Data Preprocessing → Core Analysis Pipeline: The cleaned data is then processed through the review analysis module (review_analyzer.py) where NLP tasks are applied.
• Analysis Pipeline → Metrics & Logging: The derived insights (review summaries and key metrics) are passed to the MLflow tracking system (integrated via mlops_tracker.py), which saves logs into the mlruns folder.
• Final Output → User: The final analysis results (structured JSON containing summaries and metrics) are then output and can be consumed by the user.
Architectural Patterns & Design Principles
• MLOps Integration: Clearly illustrate how MLflow is interwoven throughout the process for performance tracking, ensuring the reproducibility and continuous monitoring of experiments.
• Modular Design: Emphasize modularity with clearly separated modules for loading, processing, analysis, and logging so that each component can be improved independently.