# aydiegithub/nyc-taxi-demand-prediction architecture

> Architecture diagram of the GitHub repository aydiegithub/nyc-taxi-demand-prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aydiegithub/nyc-taxi-demand-prediction
- Repository: https://github.com/aydiegithub/nyc-taxi-demand-prediction

## Overview

##### Project Type and Purpose
- This is a data-driven, full-stack ML project with three major phases:
- Batch data processing & feature engineering (Jupyter notebooks + Python scripts)
- Model training & storage (notebooks + serialized artifacts)
- A simple prediction service (Flask API + HTML frontend)
- Purpose: Ingest historical taxi trips, preprocess and engineer features, train a regression model, and expose a realtime/near-realtime prediction endpoint for taxi demand.

##### File Structure
- DataSets/ → raw CSVs and RTF describing the original datasets
- Training_And_testing/ → exploratory notebooks & Python scripts for data prep and model training
- maps/ → generated HTML maps for cluster visualizations (output artifacts)
- main.py → orchestrates end-to-end pipeline (data prep → feature engineering → model training)
- model/ → Flask service (app.py) + clusters.csv + templates/index.html

##### README for Insights
- Confirms a Python/Jupyter-based workflow using Polars, Pandas, Scikit-learn, XGBoost
- Details preprocessing steps, clustering, lag features, and target labeling
- Mentions a “planned” Flask deployment with a basic frontend

a. Data Ingestion & Storage
- Raw data in DataSets/
- Versioned CSVs, clusters.csv, metadata
b. Data Preparation & Feature Engineering
- Jupyter notebooks (DataPreparation.ipynb) & supporting Python modules
c. Model Training
- Notebooks (ModelTraining.ipynb) producing a trained RandomForestRegressor (or XGBoost)
- Outputs: serialized model file, evaluation metrics
d. Model Registry / Artifact Store
- Location where you store model.pkl or joblib file plus clusters.csv
e. Prediction Service
- Flask app (model/app.py) loads the model + clusters.csv at startup
- Exposes REST endpoint(s) for incoming feature payloads → returns demand prediction
f. Frontend Interface
- Static HTML (model/templates/index.html) calling the Flask API
- Allows users to enter location/time parameters and view results
g. Visualization Module (optional)
- maps/ folder as a downstream consumer of clustering output—renders HTML maps

##### Architectural Patterns & Technologies
- Batch Pipeline Pattern: separate preprocessing, feature store, training
- Model serving via lightweight microservice (Flask)
- No formal DB—uses flat files/CSV as data stores
- Key tech: Python, Jupyter, Polars/Pandas, Scikit-learn/XGBoost, Flask, HTML/CSS

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Batch Processing Pipeline
    subgraph "Batch Processing Pipeline"
        DS["DataSets<br/>(raw CSV)"]:::dataStore
        DP["DataPreparation.ipynb"]:::compute
        MT["ModelTraining.ipynb"]:::compute
        MP["main.py<br/>(Orchestrator)"]:::compute
    end

    DS -->|"CSV files"| DP
    DP -->|"Features (Parquet)"| MT
    MT -->|"Model + clusters.csv"| MP

    %% Model Registry / Artifact Store
    subgraph "Model Registry / Artifact Store"
        CR1["clusters.csv"]:::dataStore
        MR["model.pkl"]:::dataStore
    end

    MP -->|"store clusters"| CR1
    MP -->|"store model"| MR

    %% Prediction Service
    subgraph "Prediction Service"
        API["Flask API<br/>(app.py)"]:::service
        CL["clusters.csv"]:::dataStore
        LM["Load model.pkl"]:::service
    end

    MR --> LM
    CR1 --> API
    LM --> API

    %% Frontend Interface
    subgraph "Frontend Interface"
        FE["index.html"]:::frontend
    end

    FE -->|"/predict"| API
    API -->|"JSON result"| FE

    %% Visualization Module
    subgraph "Visualization Module"
        VM["maps/"]:::frontend
    end

    MP -->|"cluster outputs"| VM

    %% Click Events
    click DS "https://github.com/aydiegithub/nyc-taxi-demand-prediction/tree/main/DataSets/"
    click DP "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/Training_And_testing/DataPreparation.ipynb"
    click MT "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/Training_And_testing/ModelTraining.ipynb"
    click MP "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/main.py"
    click CR1 "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/Training_And_testing/clusters.csv"
    click CL "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/model/clusters.csv"
    click API "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/model/app.py"
    click FE "https://github.com/aydiegithub/nyc-taxi-demand-prediction/blob/main/model/templates/index.html"
    click VM "https://github.com/aydiegithub/nyc-taxi-demand-prediction/tree/main/maps/"

    %% Styles
    classDef dataStore fill:#ADD8E6,stroke:#0000FF
    classDef compute fill:#90EE90,stroke:#008000
    classDef service fill:#FFA500,stroke:#FF8C00
    classDef frontend fill:#D8BFD8,stroke:#800080
```
