How marwandebbiche/post-tuto-deployment works
This repository is an end-to-end ML product that includes both offline pipelines (data collection + model training) and an online application (web UI + REST API + database) that serves a sentiment model in “real time” while users type reviews, then stores submitted reviews.
From the tree + README, the system is best represented as:
• Offline / batch: Scrape reviews → prepare dataset → train PyTorch model → export model artifact(s)
• Online / serving: Dash UI → API → ML inference → Postgres persistence
Use these directories as “truth” for components:
Online services (containerized)
• Dash frontend: src/dash/
• Entrypoint: src/dash/app.py
• Configuration: src/dash/config.py
• Static assets: src/dash/assets/
• Local CSVs for demo/data: src/dash/csv/
• Docker image: src/dash/Dockerfile, deps: src/dash/requirements.txt
• REST API service: src/api/
• Entrypoint: src/api/app.py
• Configuration: src/api/config.py
• Database access: src/api/db.py
• ML inference package: src/api/ml/
• Model wrapper/inference: src/api/ml/model.py
• Utilities: src/api/ml/utils.py
• Model files directory: src/api/ml/models/ (where trained artifacts are expected)
• Docker image: src/api/Dockerfile, deps: src/api/requirements.txt
• Database: Postgres (not in code, but defined in runtime)
• Mentioned in README as Docker-run or via Compose
• Model it as a first-class container/service
Offline pipeline (not necessarily deployed with the app)
• Scraping: src/scraping/
• Scrapy project (Trustpilot): src/scraping/scrapy/trustpilot/ with spiders/scraper.py, pipelines.py, items.py, settings.py
• Selenium notebooks (Forbes top 100 + URLs): src/scraping/selenium/*.ipynb
• Selenium driver binary: src/scraping/selenium/driver/chromedriver
• Outputs/exports: src/scraping/selenium/exports/*.csv
• Training: src/training/
• Training entrypoint: src/training/train.py
• Training library code: src/training/src/
• Data loading: data_loader.py
• Model definition: model.py
• Loss: focal_loss.py
• Utilities: utils.py
• Output locations: src/training/models/, logs: src/training/logs/
Orchestration / deployment glue
• docker-compose.yml orchestrates API + Dash (+ Postgres)
• .env indicates environment-based configuration (DB creds, API URLs, etc.)
• README indicates deployment to AWS (likely EC2 hosting Docker Compose)
User Browser
• interacts with Dash over HTTP
Dash Web App (Container)
• serves UI
• sends user text to API for inference (likely HTTP/JSON)
• submits final review + rating to API
API Service (Container)
• exposes REST endpoints (in src/api/app.py)
• routes:
• “predict sentiment” endpoint → calls ML inference module (src/api/ml/model.py)
• “create/retrieve reviews” endpoints → calls DB layer (src/api/db.py)
ML Inference Module (inside API container)
• loads model artifact(s) from src/api/ml/models/
• runs inference on incoming text
• returns sentiment score + suggested 1–5 rating
Postgres (Container)
• stores submitted reviews/ratings and (possibly) brand/company metadata
• Browser → Dash: “HTTP GET/POST (UI)”
• Dash → API: “REST /predict (text payload)”
• API → ML module: “in-process call (inference)”
• API ↔ Postgres: “SQL (insert/select reviews)”
• API → Dash: “JSON (sentiment score, suggested rating; review lists)”
Selenium Scraper (Notebooks)
• sources: Forbes top companies, websites
• outputs: CSV lists of companies/URLs (exports/*.csv)
Scrapy Spider (Trustpilot)
• pulls reviews at scale
Dataset Storage (Files)
• As an artifact store (local filesystem in repo, or S3 in cloud)
• feeds training
Training Job (PyTorch)
• train.py + training/src/*
• outputs model artifact(s) to src/training/models/
Model Promotion / Packaging Step
• “copy/export trained model → API image”
• concretely: model files land in src/api/ml/models/ (that directory exists explicitly for serving)
• Selenium → “Company URLs CSV”
• Scrapy → “Raw reviews dataset”
• Dataset → Training → “Trained model artifact”
• Trained model artifact → API container image (or mounted volume) → ML inference module
This is critical: it shows the *supply chain* of the model used by production inference.
• Service separation: Dash UI and API are independent deployable services (distinct Dockerfiles).
• 3-tier online app: UI → API → DB.
• In-process model serving: No separate model server; inference runs within the API process using model artifacts.
• Offline/online split (MLOps-lite): scraping + training are not part of the runtime path but produce artifacts consumed by runtime.
• Container orchestration via Docker Compose: single-host multi-container setup (with future suggestion of ECS/K8s).
Don’t overthink it: maximize component separation (what to explicitly split)
To be “very detailed and accurate,” split these even if they could be merged:
• Split Dash UI from API (different Dockerfiles; different concerns)
• Split API into Routing, Inference, Persistence
• Split offline into Selenium discovery vs Scrapy review harvesting
• Split storage into DB (online) vs Dataset/Artifacts (offline)