# marwandebbiche/post-tuto-deployment architecture

> Architecture diagram of the GitHub repository marwandebbiche/post-tuto-deployment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marwandebbiche/post-tuto-deployment
- Repository: https://github.com/marwandebbiche/post-tuto-deployment

## Overview

This repository is an **end-to-end ML product** that includes both **offline pipelines** (data collection + model training) and an **online application** (web UI + REST API + database) that serves a **sentiment model** in “real time” while users type reviews, then stores submitted reviews.

From the tree + README, the system is best represented as:
- **Offline / batch**: Scrape reviews → prepare dataset → train PyTorch model → export model artifact(s)
- **Online / serving**: Dash UI → API → ML inference → Postgres persistence

Use these directories as “truth” for components:

##### Online services (containerized)
- **Dash frontend**: `src/dash/`
- Entrypoint: `src/dash/app.py`
- Configuration: `src/dash/config.py`
- Static assets: `src/dash/assets/`
- Local CSVs for demo/data: `src/dash/csv/`
- Docker image: `src/dash/Dockerfile`, deps: `src/dash/requirements.txt`

- **REST API service**: `src/api/`
- Entrypoint: `src/api/app.py`
- Configuration: `src/api/config.py`
- Database access: `src/api/db.py`
- ML inference package: `src/api/ml/`
- Model wrapper/inference: `src/api/ml/model.py`
- Utilities: `src/api/ml/utils.py`
- Model files directory: `src/api/ml/models/` (where trained artifacts are expected)
- Docker image: `src/api/Dockerfile`, deps: `src/api/requirements.txt`

- **Database**: Postgres (not in code, but defined in runtime)
- Mentioned in README as Docker-run or via Compose
- Model it as a first-class container/service

##### Offline pipeline (not necessarily deployed with the app)
- **Scraping**: `src/scraping/`
- Scrapy project (Trustpilot): `src/scraping/scrapy/trustpilot/` with `spiders/scraper.py`, `pipelines.py`, `items.py`, `settings.py`
- Selenium notebooks (Forbes top 100 + URLs): `src/scraping/selenium/*.ipynb`
- Selenium driver binary: `src/scraping/selenium/driver/chromedriver`
- Outputs/exports: `src/scraping/selenium/exports/*.csv`

- **Training**: `src/training/`
- Training entrypoint: `src/training/train.py`
- Training library code: `src/training/src/`
- Data loading: `data_loader.py`
- Model definition: `model.py`
- Loss: `focal_loss.py`
- Utilities: `utils.py`
- Output locations: `src/training/models/`, logs: `src/training/logs/`

##### Orchestration / deployment glue
- `docker-compose.yml` orchestrates API + Dash (+ Postgres)
- `.env` indicates environment-based configuration (DB creds, API URLs, etc.)
- README indicates deployment to AWS (likely EC2 hosting Docker Compose)

##### User Browser
- interacts with Dash over HTTP

##### Dash Web App (Container)
- serves UI
- sends user text to API for inference (likely HTTP/JSON)
- submits final review + rating to API

##### API Service (Container)
- exposes REST endpoints (in `src/api/app.py`)
- routes:
- “predict sentiment” endpoint → calls ML inference module (`src/api/ml/model.py`)
- “create/retrieve reviews” endpoints → calls DB layer (`src/api/db.py`)

##### ML Inference Module (inside API container)
- loads model artifact(s) from `src/api/ml/models/`
- runs inference on incoming text
- returns sentiment score + suggested 1–5 rating

##### Postgres (Container)
- stores submitted reviews/ratings and (possibly) brand/company metadata

- Browser → Dash: “HTTP GET/POST (UI)”
- Dash → API: “REST /predict (text payload)”
- API → ML module: “in-process call (inference)”
- API ↔ Postgres: “SQL (insert/select reviews)”
- API → Dash: “JSON (sentiment score, suggested rating; review lists)”

##### Selenium Scraper (Notebooks)
- sources: Forbes top companies, websites
- outputs: CSV lists of companies/URLs (`exports/*.csv`)

##### Scrapy Spider (Trustpilot)
- pulls reviews at scale

##### Dataset Storage (Files)
- As an artifact store (local filesystem in repo, or S3 in cloud)
- feeds training

##### Training Job (PyTorch)
- `train.py` + `training/src/*`
- outputs model artifact(s) to `src/training/models/`

##### Model Promotion / Packaging Step
- “copy/export trained model → API image”
- concretely: model files land in `src/api/ml/models/` (that directory exists explicitly for serving)

- Selenium → “Company URLs CSV”
- Scrapy → “Raw reviews dataset”
- Dataset → Training → “Trained model artifact”
- Trained model artifact → API container image (or mounted volume) → ML inference module

This is critical: it shows the *supply chain* of the model used by production inference.

- **Service separation**: Dash UI and API are independent deployable services (distinct Dockerfiles).
- **3-tier online app**: UI → API → DB.
- **In-process model serving**: No separate model server; inference runs within the API process using model artifacts.
- **Offline/online split (MLOps-lite)**: scraping + training are not part of the runtime path but produce artifacts consumed by runtime.
- **Container orchestration via Docker Compose**: single-host multi-container setup (with future suggestion of ECS/K8s).

##### Don’t overthink it: maximize component separation (what to explicitly split)
To be “very detailed and accurate,” split these even if they could be merged:
- Split **Dash UI** from **API** (different Dockerfiles; different concerns)
- Split API into **Routing**, **Inference**, **Persistence**
- Split offline into **Selenium discovery** vs **Scrapy review harvesting**
- Split storage into **DB (online)** vs **Dataset/Artifacts (offline)**

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Diagram 1: Online Runtime (C4 Container View)
  %% =========================
  subgraph "Diagram 1 — Online Runtime (Serving Path)"
    direction TB

    subgraph "Deployment Target: AWS (EC2 / single host; future ECS/K8s)"
      direction TB

      User["User Browser"]:::external

      subgraph "Docker Compose Stack (single Docker network)"
        direction TB
        Compose["Docker Compose Orchestration"]:::orchestrator
        Env["Environment Variables (.env)"]:::config

        DashSvc["Dash Web App Container (UI, port 8050)"]:::container
        ApiSvc["REST API Container (HTTP/JSON)"]:::container
        Pg["Postgres Container (port 5432)"]:::database
      end
    end

    User -->|"HTTP_GET/POST(UI)"| DashSvc
    DashSvc -->|"REST_/predict(JSON:{text})"| ApiSvc
    ApiSvc -->|"JSON:{score,rating,review_lists}"| DashSvc
    ApiSvc <-->|"SQL(insert/select_reviews)"| Pg
  end

  %% =========================
  %% Diagram 3: API Component View (Inside API Container)
  %% =========================
  subgraph "Diagram 3 — API Internals (src/api)"
    direction TB

    ApiApp["API Routes / Entrypoint"]:::service
    ApiCfg["API Configuration"]:::config
    DbLayer["DB Access Layer (DAO/Repository)"]:::service

    subgraph "ML Inference (in-process; no separate model server)"
      direction TB
      MlPkg["ML Package"]:::service
      MlModel["Model Wrapper / Inference Logic"]:::service
      MlUtils["ML Utilities"]:::service
      MlModels[("Served Model Artifacts Directory")]:::artifactstore
    end

    ApiApp -->|"loads_config"| ApiCfg
    ApiApp -->|"calls_predict()"| MlModel
    MlModel -->|"uses"| MlUtils
    MlModel -->|"loads_artifact(model.pt)"| MlModels
    ApiApp -->|"calls_crud()"| DbLayer
    DbLayer -->|"SQL_driver"| Pg
  end

  ApiSvc -->|"in_process"| ApiApp

  %% =========================
  %% Diagram 2: Offline / Batch (Data + Training Lifecycle)
  %% =========================
  subgraph "Diagram 2 — Offline / Batch (Data Collection + Training + Promotion)"
    direction TB

    Forbes["Forbes Top Companies (External Source)"]:::external
    Websites["Company Websites (External Source)"]:::external
    Trustpilot["Trustpilot Reviews (External Source)"]:::external

    subgraph "Selenium Discovery (Notebooks)"
      direction TB
      SeleniumNB["Selenium Notebooks"]:::job
      SeleniumExports[("CSV Exports (companies/urls)")]:::artifactstore
      ChromeDriver["Chromedriver Binary"]:::runtime
    end

    subgraph "Scrapy Harvesting (Trustpilot)"
      direction TB
      ScrapyProj["Scrapy Project"]:::job
      ScrapySpider["Scrapy Spider (reviews)"]:::job
      ScrapyPipes["Pipelines / Items / Settings"]:::service
    end

    Dataset[("Dataset Storage (raw/processed files)")]:::artifactstore

    subgraph "Training (PyTorch)"
      direction TB
      TrainJob["Training Entrypoint"]:::job
      TrainLib["Training Library (data/model/loss/utils)"]:::service
      TrainModels[("Trained Model Artifacts")]:::artifactstore
      TrainLogs[("Training Logs")]:::artifactstore
    end

    Promote["Model Promotion / Packaging (copy to serving)"]:::job

    Forbes -->|"source_list"| SeleniumNB
    Websites -->|"crawl_targets"| SeleniumNB
    ChromeDriver -->|"drives_browser"| SeleniumNB
    SeleniumNB -->|"Artifact:CSV(company_urls)"| SeleniumExports

    Trustpilot -->|"upstream_reviews"| ScrapySpider
    SeleniumExports -->|"input:company_urls"| ScrapySpider
    ScrapyProj -->|"runs"| ScrapySpider
    ScrapySpider -->|"uses"| ScrapyPipes
    ScrapySpider -->|"Artifact:raw_reviews"| Dataset

    Dataset -->|"training_data"| TrainJob
    TrainJob -->|"uses"| TrainLib
    TrainJob -->|"Artifact:model_weights"| TrainModels
    TrainJob -->|"Artifact:logs"| TrainLogs

    TrainModels -.->|"Artifact_promotion(model.pt)"| Promote
    Promote -.->|"copy_into_image_or_volume"| MlModels
  end

  %% Cross-lane: runtime reads promoted model artifacts
  MlModels -.->|"mounted_or_baked_into_API"| ApiSvc

  %% =========================
  %% Click Events (from component_mapping)
  %% =========================
  click Compose "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/docker-compose.yml"
  click Env "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/.env"

  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/dash/"
  click ApiSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/api/"

  click ApiApp "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/app.py"
  click ApiCfg "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/config.py"
  click DbLayer "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/db.py"
  click MlPkg "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/api/ml/"
  click MlModel "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/ml/model.py"
  click MlUtils "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/ml/utils.py"
  click MlModels "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/api/ml/models/"

  click SeleniumNB "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/selenium/scrape_forbes_top_100.ipynb"
  click SeleniumNB "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/selenium/scrape_website_urls.ipynb"
  click SeleniumExports "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/scraping/selenium/exports/"
  click ChromeDriver "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/scraping/selenium/driver/chromedriver"

  click ScrapyProj "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/scraping/scrapy/"
  click ScrapySpider "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/scrapy/trustpilot/spiders/scraper.py"
  click ScrapyPipes "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/scrapy/trustpilot/pipelines.py"
  click ScrapyPipes "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/scrapy/trustpilot/items.py"
  click ScrapyPipes "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/scraping/scrapy/trustpilot/settings.py"

  click TrainJob "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/training/train.py"
  click TrainLib "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/training/src/data_loader.py"
  click TrainLib "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/training/src/model.py"
  click TrainLib "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/training/src/focal_loss.py"
  click TrainLib "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/training/src/utils.py"
  click TrainModels "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/training/models/"
  click TrainLogs "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/training/logs/"
  click Promote "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/training/"

  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/dash/app.py"
  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/dash/config.py"
  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/dash/assets/"
  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/dash/csv/"
  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/dash/Dockerfile"
  click DashSvc "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/dash/requirements.txt"

  click ApiSvc "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/api/Dockerfile"
  click ApiSvc "https://github.com/marwandebbiche/post-tuto-deployment/blob/master/src/api/requirements.txt"
  click ScrapyProj "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/scraping/"
  click TrainJob "https://github.com/marwandebbiche/post-tuto-deployment/tree/master/src/training/"

  %% =========================
  %% Styles
  %% =========================
  classDef external fill:#E5E7EB,stroke:#6B7280,color:#111827,stroke-width:1px
  classDef container fill:#DBEAFE,stroke:#1D4ED8,color:#0B1220,stroke-width:1.5px
  classDef database fill:#FEF3C7,stroke:#B45309,color:#111827,stroke-width:1.5px
  classDef artifactstore fill:#FDE68A,stroke:#B45309,color:#111827,stroke-width:1.5px
  classDef job fill:#E9D5FF,stroke:#7C3AED,color:#111827,stroke-width:1.5px
  classDef service fill:#CFFAFE,stroke:#0891B2,color:#0B1220,stroke-width:1.25px
  classDef orchestrator fill:#BBF7D0,stroke:#15803D,color:#052e16,stroke-width:1.5px
  classDef config fill:#FFE4E6,stroke:#BE123C,color:#111827,stroke-width:1.25px
  classDef runtime fill:#F3F4F6,stroke:#374151,color:#111827,stroke-width:1px
```
