# patrickbrus/retail_data_analytics architecture

> Architecture diagram of the GitHub repository patrickbrus/retail_data_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/patrickbrus/retail_data_analytics
- Repository: https://github.com/patrickbrus/retail_data_analytics

## Overview

##### Project Type and Purpose
- This is a machine-learning pipeline for retail data analytics (Udacity ML Engineering Capstone).
- It ingests historical retail data (stores, features, sales), explores it in notebooks, creates train/test splits, trains multiple models (Decision Tree, Linear Regression, Random Forest, XGBoost), and deploys the best model via AWS SageMaker to serve future-year sales predictions.

##### File Structure
Top-level directories and files:
- .gitignore – version-control ignore rules
- Jupyter notebooks
– 1_Data_Exploration.ipynb
– 2_Create_Train_and_Test_Data.ipynb
– 3_Training_and_Deployment.ipynb
- Documentation/ – proposal & report PDFs
- data/ – raw CSV datasets (features, sales, stores)
- data_converted/ – processed CSVs (train.csv, val.csv, test.csv, etc.)
- source/ – Python scripts + requirements.txt for training models
Patterns & clues:
- An ETL stage (notebooks 1 & 2) that reads raw data, merges and splits it.
- A training and deployment stage (notebook 3 + source scripts) that uses SageMaker SDK.
- Multiple model choices and hyperparameter tuning (scikit-optimize).
- AWS SageMaker as the execution and hosting environment.

##### Key README Insights
- Domain: predict next-year department sales per store.
- Pipeline: EDA → data engineering → model training & deployment.
- Tech stack: Python, Jupyter, scikit-learn, seaborn, matplotlib, scikit-optimize, AWS SageMaker.

##### Main Components
a. Data Layer
- Raw Data Storage (data/*.csv)
- Processed Data Storage (data_converted/*.csv)
b. Processing Layer (ETL & EDA)
- Notebook 1: Data Exploration
- Notebook 2: Data Merge & Split
c. Model Training Layer
- source/train_*.py scripts (Decision Tree, Linear Regression, Random Forest, XGBoost)
- Hyperparameter tuning (scikit-optimize)
- SageMaker training jobs
d. Model Registry & Deployment
- SageMaker Model artifacts (S3)
- SageMaker Endpoint for real-time inference
e. Client / Consumer
- API or notebook that queries the SageMaker endpoint for predictions

##### Architectural Patterns and Design Principles
- Modular pipeline: clear separation of EDA, ETL, training, deployment
- Data-centric flow: raw data → transformed data → models → predictions
- Reproducibility: notebooks plus source scripts with requirements.txt
- Cloud-native ML: leveraging SageMaker for scaling and endpoint hosting

##### Technologies, Frameworks, and Libraries
- Jupyter Notebooks (data exploration & ETL)
- Python libraries: scikit-learn, matplotlib, seaborn, scikit-optimize, AWS SageMaker SDK
- AWS SageMaker (training jobs, model registry, serving endpoints)
- S3 (for storing training data and model artifacts)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Layer"
        direction TB
        RD1["Features_data_set.csv"]:::data
        RD2["sales_data_set.csv"]:::data
        RD3["stores_data_set.csv"]:::data

        subgraph "Processed Data Storage"
            direction TB
            P1["train.csv"]:::data
            P2["val.csv"]:::data
            P3["test.csv"]:::data
            P4["test_with_true_label.csv"]:::data
            P5["features_list.csv"]:::data
        end
    end

    subgraph "Processing Layer"
        direction TB
        N1["Notebook 1: Data Exploration"]:::process
        N2["Notebook 2: Create Train/Test"]:::process
        N3["Notebook 3: Training & Deployment"]:::process
    end

    subgraph "Model Training Layer"
        direction TB
        DT["Decision Tree Script"]:::compute
        LR["Linear Regression Script"]:::compute
        RF["Random Forest Script"]:::compute
        XG["XGBoost Script"]:::compute
        REQ["requirements.txt"]:::compute
    end

    subgraph "Deployment Layer"
        direction TB
        SMTrain["SageMaker Training Job"]:::compute
        S3Models["Model Artifacts (S3)"]:::data
        Endpoint["SageMaker Endpoint"]:::service
    end

    subgraph "Client / Consumption"
        direction TB
        Client["Client Application"]:::service
        Pred["Predictions"]:::service
    end

    subgraph "Documentation"
        direction TB
        Doc1["Project Proposal"]:::docs
        Doc2["Project Report"]:::docs
        Readme["README.md"]:::docs
    end

    %% Data Flow
    RD1 -->|reads| N1
    RD2 -->|reads| N1
    RD3 -->|reads| N1
    N1 -->|"output to"| N2
    N2 -->|writes| P1
    N2 -->|writes| P2
    N2 -->|writes| P3
    N2 -->|writes| P4
    N2 -->|writes| P5

    %% Training Flow
    P1 -->|input| DT
    P2 -->|input| DT
    P3 -->|input| DT
    P4 -->|input| DT
    P5 -->|features| DT
    P1 -->|input| LR
    P2 -->|input| LR
    P3 -->|input| LR
    P4 -->|input| LR
    P5 -->|features| LR
    P1 -->|input| RF
    P2 -->|input| RF
    P3 -->|input| RF
    P4 -->|input| RF
    P5 -->|features| RF
    P1 -->|input| XG
    P2 -->|input| XG
    P3 -->|input| XG
    P4 -->|input| XG
    P5 -->|features| XG

    REQ -->|dependencies| DT
    REQ -->|dependencies| LR
    REQ -->|dependencies| RF
    REQ -->|dependencies| XG

    DT -->|trains on| SMTrain
    LR -->|trains on| SMTrain
    RF -->|trains on| SMTrain
    XG -->|trains on| SMTrain

    SMTrain -->|stores artifacts| S3Models
    S3Models -->|deploys model to| Endpoint
    Client -->|requests| Endpoint
    Endpoint -->|returns| Pred

    %% Click Events
    click RD1 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data/Features_data_set.csv"
    click RD2 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data/sales_data_set.csv"
    click RD3 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data/stores_data_set.csv"
    click P1 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data_converted/train.csv"
    click P2 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data_converted/val.csv"
    click P3 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data_converted/test.csv"
    click P4 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data_converted/test_with_true_label.csv"
    click P5 "https://github.com/patrickbrus/retail_data_analytics/blob/master/data_converted/features_list.csv"
    click N1 "https://github.com/patrickbrus/retail_data_analytics/blob/master/1_Data_Exploration.ipynb"
    click N2 "https://github.com/patrickbrus/retail_data_analytics/blob/master/2_Create_Train_and_Test_Data.ipynb"
    click N3 "https://github.com/patrickbrus/retail_data_analytics/blob/master/3_Training_and_Deployment.ipynb"
    click DT "https://github.com/patrickbrus/retail_data_analytics/blob/master/source/train_decision_tree.py"
    click LR "https://github.com/patrickbrus/retail_data_analytics/blob/master/source/train_linear_regression.py"
    click RF "https://github.com/patrickbrus/retail_data_analytics/blob/master/source/train_random_forest.py"
    click XG "https://github.com/patrickbrus/retail_data_analytics/blob/master/source/train_xgboost.py"
    click REQ "https://github.com/patrickbrus/retail_data_analytics/blob/master/source/requirements.txt"
    click Doc1 "https://github.com/patrickbrus/retail_data_analytics/blob/master/Documentation/Project_Proposal.pdf"
    click Doc2 "https://github.com/patrickbrus/retail_data_analytics/blob/master/Documentation/Project_Report.pdf"
    click Readme "https://github.com/patrickbrus/retail_data_analytics/blob/master/README.md"

    %% Styles
    classDef data fill:#cfe2f3,stroke:#6fa8dc
    classDef process fill:#d9ead3,stroke:#6aa84f
    classDef compute fill:#f9cb9c,stroke:#e69138
    classDef service fill:#c9c,stroke:#966
    classDef docs fill:#eeeeee,stroke:#999
```
