How aydiegithub/nyc-taxi-demand-prediction works
Project Type and Purpose
• This is a data-driven, full-stack ML project with three major phases:
• Batch data processing & feature engineering (Jupyter notebooks + Python scripts)
• Model training & storage (notebooks + serialized artifacts)
• A simple prediction service (Flask API + HTML frontend)
• Purpose: Ingest historical taxi trips, preprocess and engineer features, train a regression model, and expose a realtime/near-realtime prediction endpoint for taxi demand.
File Structure
• DataSets/ → raw CSVs and RTF describing the original datasets
• Training_And_testing/ → exploratory notebooks & Python scripts for data prep and model training
• maps/ → generated HTML maps for cluster visualizations (output artifacts)
• main.py → orchestrates end-to-end pipeline (data prep → feature engineering → model training)
• model/ → Flask service (app.py) + clusters.csv + templates/index.html
README for Insights
• Confirms a Python/Jupyter-based workflow using Polars, Pandas, Scikit-learn, XGBoost
• Details preprocessing steps, clustering, lag features, and target labeling
• Mentions a “planned” Flask deployment with a basic frontend
a. Data Ingestion & Storage
• Raw data in DataSets/
• Versioned CSVs, clusters.csv, metadata
b. Data Preparation & Feature Engineering
• Jupyter notebooks (DataPreparation.ipynb) & supporting Python modules
c. Model Training
• Notebooks (ModelTraining.ipynb) producing a trained RandomForestRegressor (or XGBoost)
• Outputs: serialized model file, evaluation metrics
d. Model Registry / Artifact Store
• Location where you store model.pkl or joblib file plus clusters.csv
e. Prediction Service
• Flask app (model/app.py) loads the model + clusters.csv at startup
• Exposes REST endpoint(s) for incoming feature payloads → returns demand prediction
f. Frontend Interface
• Static HTML (model/templates/index.html) calling the Flask API
• Allows users to enter location/time parameters and view results
g. Visualization Module (optional)
• maps/ folder as a downstream consumer of clustering output—renders HTML maps
Architectural Patterns & Technologies
• Batch Pipeline Pattern: separate preprocessing, feature store, training
• Model serving via lightweight microservice (Flask)
• No formal DB—uses flat files/CSV as data stores
• Key tech: Python, Jupyter, Polars/Pandas, Scikit-learn/XGBoost, Flask, HTML/CSS