How sebastianmunozcodes/nyc-311-data-pipeline works
This repository is a PySpark data-engineering pipeline for NYC 311 service requests. Its main workflow ingests raw CSV, inspects data quality, cleans records into Parquet, derives request and resolution features plus summary datasets, and runs Spark SQL analysis. The README also documents a Databricks workflow for loading the transformed dataset and reproducing analysis and visualizations. The Databricks implementation is unsampled, so its connections are based on README statements; other edges cite the source or README evidence shown.
Main entry points: ingest.py CSV Ingestion · transform.py Feature Transformation · transformed Transformed Parquet · clean.py Cleaning
Components 13
Local Processing
- CSV Ingestion
src/ingest.py - Quality Inspection
src/inspection.py - Cleaning
src/clean.py - Feature Transformation
src/transform.py
Data Products
- Raw CSV
data/raw - Cleaned Parquet
data/processed - Transformed Parquet
data/transformed - Summary Parquets
data/transformed
SQL Analysis
- SQL Execution
src/run_sql.py - Analytical Queries
sql/analysis.sql
Databricks Analysis
- Databricks Notebook
databricks/nyc_311_analysis.ipynb - Visual Analysis
databricks/nyc_311_analysis.html
Other components
- NYC Open Data
Connections 13
- NYC Open Data → Raw CSV: provides data
README.md - Raw CSV → CSV Ingestion: reads CSV
src/ingest.py - CSV Ingestion → Quality Inspection: loads DataFrame
src/inspection.py - CSV Ingestion → Cleaning: loads DataFrame
src/clean.py - Quality Inspection → Cleaning: informs rules
README.md - Cleaning → Cleaned Parquet: writes Parquet
src/clean.py - Cleaned Parquet → Feature Transformation: reads Parquet
src/transform.py - Feature Transformation → Transformed Parquet: writes features
src/transform.py - Feature Transformation → Summary Parquets: writes summaries
src/transform.py - Transformed Parquet → SQL Execution: registers view
src/run_sql.py - SQL Execution → Analytical Queries: executes queries
README.md - Transformed Parquet → Databricks Notebook: loads dataset
README.md - Databricks Notebook → Visual Analysis: produces visuals
README.md