How joedkhar/analysing-nyc-taxi-trips works
Project type and purpose
• This is a data-analysis/ETL pipeline with an optional interactive dashboard (app.py).
• Purpose: Ingest NYC taxi trip data (Parquet), convert and preprocess it, perform statistical analysis (Python + R), and deliver interactive visualizations and static reports (PDFs).
File structure
Top-level files & folders:
• app.py → likely a Flask/Dash/FastAPI app for visualization
• pconvertcsv.ipynb & pconvertcsv.py → Parquet→CSV conversion
• pconp.py → data preprocessing & sampling
• rTest.R / rTest2222.R → statistical analysis in R
• assets/ → static assets (background image for dashboard)
• dataset/ → raw & intermediate CSV files (hourly_taxi_stats.csv, test2.csv)
• pdfs/ → final reports, plots, inferences, poster
• requirements.txt → Python dependencies
No microservices or frontend/backend split; this is a single repo containing the entire pipeline.
README for additional insights
• Confirms the three-stage pipeline: Convert → Preprocess → R Analysis
• Output files: converted.csv, hourly_taxi_stats.csv, sampled_taxi_data.csv, test2.csv
• Technologies: Python 3.7+, Pandas, PyArrow, NumPy; R 4.0+, ggplot2, dplyr, lubridate
Main components & relationships
a. Data Sources (External)
• Original Parquet file (yellow_tripdata_2024-09.parquet) – not in repo due to size
b. ETL Layer (Python Scripts)
1. Conversion: pconvertcsv.py / notebook
Input: Parquet
Output: converted.csv
2. Preprocessing: pconp.py
Input: converted.csv
Output: sampled_taxi_data.csv, hourly_taxi_stats.csv, test2.csv
c. Statistical Analysis Layer (R Scripts)
• rTest.R, rTest2222.R
• Inputs: the CSVs from ETL
• Outputs: PDF reports in pdfs/
d. Visualization / Dashboard
• app.py + assets/
• Reads processed CSVs and serves interactive charts
e. Documentation
• pdfs/ contains static deliverables
Architectural patterns & design principles
• Simple linear ETL pipeline
• Clear separation of concerns:
• Data ingestion/conversion
• Data transformation/preprocessing
• Statistical analysis
• Presentation layer (static + interactive)
• Use of different languages/tools for tasks they excel in: Python for ETL, R for statistics