# joedkhar/analysing-nyc-taxi-trips architecture

> Architecture diagram of the GitHub repository joedkhar/analysing-nyc-taxi-trips, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joedkhar/analysing-nyc-taxi-trips
- Repository: https://github.com/joedkhar/analysing-nyc-taxi-trips

## Overview

##### Project type and purpose
- This is a data-analysis/ETL pipeline with an optional interactive dashboard (app.py).
- Purpose: Ingest NYC taxi trip data (Parquet), convert and preprocess it, perform statistical analysis (Python + R), and deliver interactive visualizations and static reports (PDFs).

##### File structure
Top-level files & folders:
- app.py            → likely a Flask/Dash/FastAPI app for visualization
- pconvertcsv.ipynb & pconvertcsv.py → Parquet→CSV conversion
- pconp.py          → data preprocessing & sampling
- rTest.R / rTest2222.R → statistical analysis in R
- assets/           → static assets (background image for dashboard)
- dataset/          → raw & intermediate CSV files (hourly_taxi_stats.csv, test2.csv)
- pdfs/             → final reports, plots, inferences, poster
- requirements.txt  → Python dependencies
No microservices or frontend/backend split; this is a single repo containing the entire pipeline.

##### README for additional insights
- Confirms the three-stage pipeline: Convert → Preprocess → R Analysis
- Output files: converted.csv, hourly_taxi_stats.csv, sampled_taxi_data.csv, test2.csv
- Technologies: Python 3.7+, Pandas, PyArrow, NumPy; R 4.0+, ggplot2, dplyr, lubridate

##### Main components & relationships
a. Data Sources (External)
- Original Parquet file (yellow_tripdata_2024-09.parquet) – not in repo due to size
b. ETL Layer (Python Scripts)
1. Conversion: pconvertcsv.py / notebook
Input: Parquet
Output: converted.csv
2. Preprocessing: pconp.py
Input: converted.csv
Output: sampled_taxi_data.csv, hourly_taxi_stats.csv, test2.csv
c. Statistical Analysis Layer (R Scripts)
- rTest.R, rTest2222.R
- Inputs: the CSVs from ETL
- Outputs: PDF reports in pdfs/
d. Visualization / Dashboard
- app.py + assets/
- Reads processed CSVs and serves interactive charts
e. Documentation
- pdfs/ contains static deliverables

##### Architectural patterns & design principles
- Simple linear ETL pipeline
- Clear separation of concerns:
- Data ingestion/conversion
- Data transformation/preprocessing
- Statistical analysis
- Presentation layer (static + interactive)
- Use of different languages/tools for tasks they excel in: Python for ETL, R for statistics

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    source["Original Parquet\n(yellow_tripdata_2024-09.parquet)"]:::external

    %% Python ETL Subgraph
    subgraph "ETL Layer (Python)"
        conv["Data Conversion\n(Parquet→CSV)"]:::script
        cs1["converted.csv"]:::datastore
        preproc["Data Preprocessing\n& Sampling"]:::script
        cs2["sampled_taxi_data.csv"]:::datastore
        cs3["hourly_taxi_stats.csv"]:::datastore
        cs4["test2.csv"]:::datastore
        requirements["requirements.txt"]:::script
    end

    %% R Analysis Subgraph
    subgraph "Statistical Analysis (R)"
        rA["rTest.R"]:::script
        rB["rTest2222.R"]:::script
    end

    %% Presentation Layer
    subgraph "Presentation"
        dashboard["Interactive Dashboard\napp.py"]:::dashboard
        assets["Static Assets\nassets/"]:::dashboard
        reports["PDF Reports\npdfs/"]:::deliverable
    end

    %% Data Flow
    source -->|converted.csv| conv
    conv --> cs1
    cs1 --> preproc
    preproc -->|sampled_taxi_data.csv| cs2
    preproc -->|hourly_taxi_stats.csv| cs3
    preproc -->|test2.csv| cs4

    preproc --> rA
    preproc --> rB
    rA --> reports
    rB --> reports

    preproc --> dashboard
    cs2 --> dashboard
    cs3 --> dashboard
    cs4 --> dashboard
    dashboard --> assets

    %% Click Events
    click conv "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/pconvertcsv.ipynb"
    click preproc "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/pconp.py"
    click rA "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/rTest.R"
    click rB "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/rTest2222.R"
    click dashboard "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/app.py"
    click assets "https://github.com/joedkhar/analysing-nyc-taxi-trips/tree/main/assets/"
    click cs3 "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/dataset/dataset/hourly_taxi_stats.csv"
    click cs4 "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/dataset/dataset/test2.csv"
    click reports "https://github.com/joedkhar/analysing-nyc-taxi-trips/tree/main/pdfs/"
    click requirements "https://github.com/joedkhar/analysing-nyc-taxi-trips/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#FFEB3B,stroke:#333,stroke-width:1px;
    classDef datastore fill:#ECEFF1,stroke:#555,stroke-width:1px;
    classDef script fill:#BBDEFB,stroke:#1E88E5,stroke-width:1px;
    classDef dashboard fill:#C8E6C9,stroke:#2E7D32,stroke-width:1px;
    classDef deliverable fill:#FFE0B2,stroke:#FB8C00,stroke-width:1px;
```
