# szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics architecture

> Architecture diagram of the GitHub repository szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics
- Repository: https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics

## Overview

##### Project type and purpose
- This is an end-to-end data ingestion, ELT, warehousing, and machine-learning pipeline for e-commerce analytics. It uses Dagster for orchestration, Python scripts for extraction/cleaning, dbt for transformations and tests, BigQuery as the cloud data warehouse, and a two-step clustering ML pipeline (SOM + K-Means) for customer segmentation.

##### File structure
- Top-level files:
- start_dagster.sh – wrapper script to launch Dagster and dbt.
- environment.yml / pyproject.toml – dependency definitions.
- README.md, docs/, assets/, notebooks/.
- olist_ecommerce_orchestration/olist_ecommerce/ – a full dbt project:
- models/staging/ – SQL to stage raw tables.
- models/marts/ – star-schema marts (core, sales, finance, operations, customer_service).
- seeds/ – static lookup data.
- snapshots/ – snapshot definitions for slowly changing dimensions.
- dbt_project.yml, profiles.yml – dbt config.
- olist_ecommerce_orchestration/olist_ecommerce_orchestration/ – Dagster code:
- definitions.py – job and asset definitions.
- io_managers.py – I/O management (BigQuery, local).
- scripts/ – Python modules: Kaggle dataset download, data cleaning, BQ uploads, key management.
- notebooks/ml_model_customer_segmentation.ipynb – exploratory & production notebook for ML pipeline.

##### README for additional insights
- The README lays out three primary jobs:
a. python_elt_job – handles data ingestion, cleaning, loading into BigQuery, then runs dbt to build staging and marts.
b. som_kmeans_segmentation_pipeline – depends on dbt marts, performs feature engineering, trains SOM + K-Means, materializes segmented tables.
c. full_data_ml_pipeline – orchestrates both ELT and ML in one end-to-end run.
- Cloud services: Google BigQuery, GCS (implied by GCS bucket env var), Kaggle API.
- Visualizations in assets/ illustrate asset lineage and SOM maps.

##### Elements
- Blue = Data storage (BigQuery)
- Green = Processing (Python scripts, dbt)
- Orange = Orchestration (Dagster jobs)
- Purple = Machine Learning components
- Annotations for environment/config files (environment.yml, start_dagster.sh) and external services (Kaggle API, GCP credentials).

##### Final recommendations
- Essential architectural layers (Ingestion, Load, Transform, Orchestration, ML).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        Kag["Kaggle CSVs"]:::external
        click Kag "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/scripts/load_kaggle_dataset.py"
        KeyMan["GCP Key Management"]:::processing
        click KeyMan "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/scripts/move_key_file.py"
        Clean["Python Data Cleaning"]:::processing
        click Clean "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/scripts/data_cleaning_utils.py"
    end

    %% Loading Layer
    subgraph "Loading"
        BQLoad["BigQuery Batch Upload"]:::processing
        click BQLoad "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/scripts/bq_upload_batch.py"
        Raw["BigQuery Raw Tables"]:::storage
    end

    %% Transformation Layer
    subgraph "Transformation (dbt)"
        Sources["dbt Source Definitions"]:::processing
        click Sources "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce/models/sources.yml"
        Staging["dbt Staging Models"]:::processing
        click Staging "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/tree/main/olist_ecommerce_orchestration/olist_ecommerce/models/staging/"
        Marts["dbt Mart Models"]:::processing
        click Marts "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/tree/main/olist_ecommerce_orchestration/olist_ecommerce/models/marts/"
        Snapshots["dbt Snapshots"]:::processing
        click Snapshots "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/tree/main/olist_ecommerce_orchestration/olist_ecommerce/snapshots/"
        Seeds["dbt Seeds"]:::processing
        click Seeds "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/tree/main/olist_ecommerce_orchestration/olist_ecommerce/seeds/"
        Config["dbt Configurations"]:::processing
        click Config "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce/dbt_project.yml"
        click Config "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce/profiles.yml"
    end

    %% Machine Learning Layer
    subgraph "Machine Learning"
        Notebook["ML Notebook (SOM + K-Means)"]:::ml
        click Notebook "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/notebooks/ml_model_customer_segmentation.ipynb"
        FE["Feature Engineering"]:::ml
        SOM["SOM Training & Visualization"]:::ml
        KMeans["K-Means Clustering"]:::ml
    end

    %% Results
    Segmented["BigQuery Segmented Tables"]:::storage

    %% Orchestration Layer
    subgraph "Orchestration (Dagster)"
        IO["Dagster I/O Managers"]:::orchestration
        click IO "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/io_managers.py"
        Defs["Dagster Job & Asset Definitions"]:::orchestration
        click Defs "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/olist_ecommerce_orchestration/olist_ecommerce_orchestration/definitions.py"
        Entry["ELT Entrypoint Script"]:::orchestration
        click Entry "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/start_dagster.sh"
        Env["Env & Dependency Config"]:::external
        click Env "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/environment.yml"
        click Env "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/blob/main/pyproject.toml"
        Docs["Documentation & Assets"]:::external
        click Docs "https://github.com/szevy/end-to-end-data-ml-pipeline-for-ecommerce-analytics/tree/main/docs/"
    end

    %% Data Flow
    Kag --> Clean --> BQLoad --> Raw
    KeyMan --> BQLoad
    Raw --> Sources --> Staging --> Marts --> FE --> SOM --> KMeans --> Segmented
    Raw --> Snapshots
    Raw --> Seeds

    %% Orchestration Flow
    Entry --> Defs
    Defs --> IO
    Defs --> Clean
    Defs --> BQLoad
    Defs --> Staging
    Defs --> Marts
    Defs --> FE
    Defs --> SOM
    Defs --> KMeans

    %% Styles
    classDef storage fill:#bbdefb,stroke:#0d47a1,color:#0d47a1;
    classDef processing fill:#c8e6c9,stroke:#2e7d32,color:#2e7d32;
    classDef orchestration fill:#ffe0b2,stroke:#ef6c00,color:#ef6c00;
    classDef ml fill:#e1bee7,stroke:#6a1b9a,color:#6a1b9a;
    classDef external fill:#f5f5f5,stroke:#9e9e9e,color:#616161;
```
