How patrickbrus/retail_data_analytics works
Project Type and Purpose
• This is a machine-learning pipeline for retail data analytics (Udacity ML Engineering Capstone).
• It ingests historical retail data (stores, features, sales), explores it in notebooks, creates train/test splits, trains multiple models (Decision Tree, Linear Regression, Random Forest, XGBoost), and deploys the best model via AWS SageMaker to serve future-year sales predictions.
File Structure
Top-level directories and files:
• .gitignore – version-control ignore rules
• Jupyter notebooks
– 1_Data_Exploration.ipynb
– 2_Create_Train_and_Test_Data.ipynb
– 3_Training_and_Deployment.ipynb
• Documentation/ – proposal & report PDFs
• data/ – raw CSV datasets (features, sales, stores)
• data_converted/ – processed CSVs (train.csv, val.csv, test.csv, etc.)
• source/ – Python scripts + requirements.txt for training models
Patterns & clues:
• An ETL stage (notebooks 1 & 2) that reads raw data, merges and splits it.
• A training and deployment stage (notebook 3 + source scripts) that uses SageMaker SDK.
• Multiple model choices and hyperparameter tuning (scikit-optimize).
• AWS SageMaker as the execution and hosting environment.
Key README Insights
• Domain: predict next-year department sales per store.
• Pipeline: EDA → data engineering → model training & deployment.
• Tech stack: Python, Jupyter, scikit-learn, seaborn, matplotlib, scikit-optimize, AWS SageMaker.
Main Components
a. Data Layer
• Raw Data Storage (data/*.csv)
• Processed Data Storage (data_converted/*.csv)
b. Processing Layer (ETL & EDA)
• Notebook 1: Data Exploration
• Notebook 2: Data Merge & Split
c. Model Training Layer
• source/train_*.py scripts (Decision Tree, Linear Regression, Random Forest, XGBoost)
• Hyperparameter tuning (scikit-optimize)
• SageMaker training jobs
d. Model Registry & Deployment
• SageMaker Model artifacts (S3)
• SageMaker Endpoint for real-time inference
e. Client / Consumer
• API or notebook that queries the SageMaker endpoint for predictions
Architectural Patterns and Design Principles
• Modular pipeline: clear separation of EDA, ETL, training, deployment
• Data-centric flow: raw data → transformed data → models → predictions
• Reproducibility: notebooks plus source scripts with requirements.txt
• Cloud-native ML: leveraging SageMaker for scaling and endpoint hosting
Technologies, Frameworks, and Libraries
• Jupyter Notebooks (data exploration & ETL)
• Python libraries: scikit-learn, matplotlib, seaborn, scikit-optimize, AWS SageMaker SDK
• AWS SageMaker (training jobs, model registry, serving endpoints)
• S3 (for storing training data and model artifacts)