How shapniljoy/delivery-time-prediction works
1. Repository purpose
- This is a Python-based machine learning research repo for predicting Swiggy food delivery time.
- The repo is organized around an offline experimentation workflow rather than a running application or API.
2. Primary data boundary
- The dataset is managed as a versioned artifact with DVC.
- data/raw/swiggy.csv.dvc points to the raw source data, while data/processed/cleaned_swiggy.csv holds the cleaned modeling dataset.
- data/interim, data/external, and data/processed separate raw inputs, intermediate transformations, and final training-ready data.
3. Notebook-driven analysis and modeling
- Most of the active experimentation lives in notebooks/.
- The notebook set maps closely to the ML lifecycle:
- EDA and missing-value inspection
- data cleaning and imputation
- feature selection
- baseline model selection
- hyperparameter tuning for multiple model families
- final stacked model assembly
- notebooks/Data_cleaning_utils.py is the main reusable helper outside the notebooks.
4. Modeling architecture
- The final predictive system is a stacked regression ensemble.
- Base learners are CatBoost, LightGBM, and Random Forest.
- A Lasso regression meta-learner combines their out-of-fold predictions.
- The repo also includes a smaller ANN experiment, but the tree-based ensemble is the selected final architecture.
5. Feature engineering and selection flow
- The project explicitly evaluates imputation vs row dropping and concludes that dropping missing rows performed better.
- Feature selection was explored with Random Forest importance, GBM importance, and RFE.
- The modeling set keeps the full feature set for final robustness, even though a small subset explains most importance.
6. Artifacts and outputs
- models/ is the designated home for saved model artifacts, though the tree shows it as mostly empty in source control.
- reports/EDA_Report.pdf captures the exploratory analysis output.
- reports/figures/ is reserved for generated plots and visual assets.
- notebooks/catboost_info/ contains CatBoost training logs and metrics artifacts, showing that some experiment outputs are captured directly from training runs.
7. Reusable source package
- src/ is structured as a standard Python package with logical boundaries:
- src/data for data access and preparation
- src/features for feature logic
- src/models for training/inference/model utilities
- src/visualization for plotting/reporting helpers
- These directories are mostly scaffolding in the tree, but they define the intended productionization boundary beyond notebooks.
8. Experiment tracking and reproducibility
- The README states that the final model and metrics were tracked with MLflow and DagsHub.
- This means the project treats model training as a repeatable experiment pipeline with external tracking/versioning, not just local notebook outputs.
9. Project orchestration and developer tooling
- Makefile, tox.ini, and test_environment.py provide environment and quality checks.
- requirements.txt, requirement-dev.txt, and requirement-dockers.txt indicate separate dependency sets for runtime, development, and containerized execution.
- setup.py makes the package installable as a Python project.
10. Documentation layer
- docs/ is a Sphinx documentation site, with pages for getting started and commands.
- This suggests the repo is meant to be understandable and reproducible by other engineers, not just the original notebook author.
- extract_notebooks.py and notebook_summaries.txt imply an effort to summarize notebook work into more durable documentation.
11. End-to-end data flow
- The effective pipeline is:
- raw Swiggy data enters via DVC
- cleaning and missing-value handling produce a processed CSV
- EDA and feature selection define the modeling inputs
- multiple regressors are trained and tuned
- stacked ensemble predictions are evaluated
- final metrics and artifacts are stored in reports/models/tracking systems
12. Architectural takeaway
- The repo’s main boundary is between data/experimentation and reusable ML code.
- The notebooks hold the research workflow; src/ and the artifact directories are the scaffold for a more modular, reproducible ML pipeline.
Main entry points: Final_model.ipynb Final stack · Feature Selection.ipynb Feature selection · cleaned_swiggy.csv Cleaned dataset · Data Cleaing.ipynb Cleaning
Components 24
Data & artifacts
Versioned inputs, processed datasets, and persisted experiment outputs.
- Raw Swiggy data · Versioned raw source dataset tracked through DVC.
data/raw/swiggy.csv.dvc - Interim data · Intermediate transformed data kept between raw and cleaned datasets.
data/interim - Cleaned dataset · Training-ready cleaned CSV used for modeling experiments.
data/processed/cleaned_swiggy.csv - Reports · EDA report and generated figures for sharing results.
reports - Saved models · Directory reserved for persisted model artifacts and exports.
models - MLflow / DagsHub · External tracking and versioning for runs, metrics, and model artifacts.
Notebook workflow
Exploratory analysis, cleaning, selection, tuning, and final ensemble assembly.
- EDA · Exploratory analysis and missing-value inspection.
notebooks/EDA.ipynb - Missing values · Focused inspection of missingness patterns and data quality.
notebooks/missing Values Inspection.ipynb - Cleaning · Data cleaning and transformation workflow.
notebooks/Data Cleaing.ipynb - Imputation · Comparison of imputation strategies versus dropping rows.
notebooks/Imputation.ipynb - Feature selection · Selection using importance methods and RFE.
notebooks/Feature Selection.ipynb - Baselines · Baseline model selection and early model comparisons.
notebooks/Baseline Model Selection.ipynb - Model tuning · Hyperparameter search for CatBoost, LightGBM, Random Forest, and stacking.
- ANN trial · Smaller neural-network experiment alongside tree-based models.
notebooks/ANN.ipynb - Final stack · Final stacked regression assembly and evaluation.
notebooks/Final_model.ipynb - Cleaning utils · Reusable cleaning and preprocessing helpers shared across notebooks.
notebooks/Data_cleaning_utils.py
Reusable package
Installable Python package boundaries for data, features, models, and visualization.
- src.data · Package boundary for data access and preparation code.
src/data - src.features · Feature engineering and feature logic package boundary.
src/features - src.models · Training, inference, and model utility package boundary.
src/models - src.visualization · Plotting and reporting helpers package boundary.
src/visualization
Docs & tooling
Documentation, environment setup, packaging, and project maintenance tooling.
- Docs site · Project documentation site with getting started and command guides.
docs - Makefile · Project automation for common development and workflow commands.
Makefile - Packaging · Installable project setup and dependency management entrypoint.
setup.py - Env checks · Environment and tooling validation for development reproducibility.
test_environment.py
Connections 24
- Raw Swiggy data → Interim data: stage · Raw versioned data is prepared into intermediate form.
- Interim data → Cleaning: feed · Intermediate data is consumed by cleaning workflows.
- Cleaning → Cleaned dataset: produce · Cleaning produces the training-ready CSV.
- Missing values → Imputation: inform · Missingness analysis informs imputation experiments.
- EDA → Feature selection: guide · EDA findings guide feature selection choices.
- Imputation → Feature selection: compare · Imputation is contrasted with row-dropping before selection.
- Cleaned dataset → Baselines: train · Cleaned data feeds baseline model evaluation.
- Cleaned dataset → Model tuning: train · Training data feeds hyperparameter optimization.
- Feature selection → Model tuning: shape · Selected feature set constrains tuning experiments.
- Model tuning → Final stack: assemble · Tuned base learners are combined in the final stack.
- Baselines → Final stack: compare · Baseline results are used as reference for the final model.
- ANN trial → Final stack: side experiment · ANN work runs alongside the selected ensemble path.
- Final stack → MLflow / DagsHub: log · Final metrics and artifacts are tracked externally.
- Final stack → Saved models: persist · Final model artifacts are saved under the models directory.
- Final stack → Reports: report · Final results are summarized into reports and figures.
- Cleaning utils → Cleaning: support · Reusable helper functions support notebook cleaning steps.
- Cleaning utils → Imputation: support · Shared preprocessing utilities support imputation experiments.
- src.data → Raw Swiggy data: wrap · Package-level data code is expected to operate on raw inputs.
- src.features → Feature selection: encapsulate · Feature package is the reusable home for feature logic.
- src.models → Final stack: encapsulate · Model package is the reusable home for training and inference logic.
- src.visualization → Reports: generate · Visualization helpers support report and figure generation.
- Docs site → Reports: document · Docs complement the analytical report outputs.
- Makefile → Packaging: orchestrate · Build automation and packaging support project execution.
- Env checks → Packaging: validate · Environment checks validate the installable project setup.