# sebastianmunozcodes/nyc-311-data-pipeline architecture

> Architecture diagram of the GitHub repository sebastianmunozcodes/nyc-311-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sebastianmunozcodes/nyc-311-data-pipeline
- Repository: https://github.com/sebastianmunozcodes/nyc-311-data-pipeline

## Overview

This repository is a PySpark data-engineering pipeline for NYC 311 service requests. Its main workflow ingests raw CSV, inspects data quality, cleans records into Parquet, derives request and resolution features plus summary datasets, and runs Spark SQL analysis. The README also documents a Databricks workflow for loading the transformed dataset and reproducing analysis and visualizations. The Databricks implementation is unsampled, so its connections are based on README statements; other edges cite the source or README evidence shown.

Main entry points: [`src/ingest.py`](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/ingest.py) (CSV Ingestion), [`src/transform.py`](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/transform.py) (Feature Transformation), [`data/transformed`](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/transformed) (Transformed Parquet), [`src/clean.py`](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/clean.py) (Cleaning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_processing["Local Processing"]
  node_ingestion["CSV Ingestion<br/>[ingest.py]"]
  node_inspection["Quality Inspection<br/>[inspection.py]"]
  node_cleaning["Cleaning<br/>[clean.py]"]
  node_transformation["Feature Transformation<br/>[transform.py]"]
end

subgraph group_data["Data Products"]
  node_raw_csv["Raw CSV"]
  node_cleaned_parquet[("Cleaned Parquet")]
  node_transformed_parquet[("Transformed Parquet")]
  node_summary_parquet[("Summary Parquets")]
end

subgraph group_analysis["SQL Analysis"]
  node_sql_runner["SQL Execution<br/>[run_sql.py]"]
  node_sql_queries["Analytical Queries<br/>[analysis.sql]"]
end

subgraph group_cloud["Databricks Analysis"]
  node_databricks["Databricks Notebook"]
  node_visual_analysis["Visual Analysis"]
end

node_nyc_open_data(("NYC Open Data"))

node_nyc_open_data -->|"provides data"| node_raw_csv
node_raw_csv -->|"reads CSV"| node_ingestion
node_ingestion -->|"loads DataFrame"| node_inspection
node_ingestion -->|"loads DataFrame"| node_cleaning
node_inspection -->|"informs rules"| node_cleaning
node_cleaning -->|"writes Parquet"| node_cleaned_parquet
node_cleaned_parquet -->|"reads Parquet"| node_transformation
node_transformation -->|"writes features"| node_transformed_parquet
node_transformation -->|"writes summaries"| node_summary_parquet
node_transformed_parquet -->|"registers view"| node_sql_runner
node_sql_runner -->|"executes queries"| node_sql_queries
node_transformed_parquet -->|"loads dataset"| node_databricks
node_databricks -->|"produces visuals"| node_visual_analysis

click node_raw_csv "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/raw"
click node_ingestion "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/ingest.py"
click node_inspection "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/inspection.py"
click node_cleaning "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/clean.py"
click node_cleaned_parquet "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/processed"
click node_transformation "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/transform.py"
click node_transformed_parquet "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/transformed"
click node_summary_parquet "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/transformed"
click node_sql_runner "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/run_sql.py"
click node_sql_queries "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/sql/analysis.sql"
click node_databricks "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/databricks/nyc_311_analysis.ipynb"
click node_visual_analysis "https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/databricks/nyc_311_analysis.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_inspection,node_cleaning,node_transformation toneBlue
class node_raw_csv,node_cleaned_parquet,node_transformed_parquet,node_summary_parquet toneAmber
class node_sql_runner,node_sql_queries toneMint
class node_databricks,node_visual_analysis toneRose
class node_nyc_open_data toneIndigo
```

## Components

### Local Processing

- [CSV Ingestion](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/ingest.py): `src/ingest.py`
- [Quality Inspection](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/inspection.py): `src/inspection.py`
- [Cleaning](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/clean.py): `src/clean.py`
- [Feature Transformation](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/transform.py): `src/transform.py`

### Data Products

- [Raw CSV](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/raw): `data/raw`
- [Cleaned Parquet](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/processed): `data/processed`
- [Transformed Parquet](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/transformed): `data/transformed`
- [Summary Parquets](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/tree/main/data/transformed): `data/transformed`

### SQL Analysis

- [SQL Execution](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/src/run_sql.py): `src/run_sql.py`
- [Analytical Queries](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/sql/analysis.sql): `sql/analysis.sql`

### Databricks Analysis

- [Databricks Notebook](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/databricks/nyc_311_analysis.ipynb): `databricks/nyc_311_analysis.ipynb`
- [Visual Analysis](https://github.com/sebastianmunozcodes/nyc-311-data-pipeline/blob/main/databricks/nyc_311_analysis.html): `databricks/nyc_311_analysis.html`

### Other components

- **NYC Open Data**

## Connections

- NYC Open Data → Raw CSV: provides data (evidence: `README.md`)
- Raw CSV → CSV Ingestion: reads CSV (evidence: `src/ingest.py`)
- CSV Ingestion → Quality Inspection: loads DataFrame (evidence: `src/inspection.py`)
- CSV Ingestion → Cleaning: loads DataFrame (evidence: `src/clean.py`)
- Quality Inspection → Cleaning: informs rules (evidence: `README.md`)
- Cleaning → Cleaned Parquet: writes Parquet (evidence: `src/clean.py`)
- Cleaned Parquet → Feature Transformation: reads Parquet (evidence: `src/transform.py`)
- Feature Transformation → Transformed Parquet: writes features (evidence: `src/transform.py`)
- Feature Transformation → Summary Parquets: writes summaries (evidence: `src/transform.py`)
- Transformed Parquet → SQL Execution: registers view (evidence: `src/run_sql.py`)
- SQL Execution → Analytical Queries: executes queries (evidence: `README.md`)
- Transformed Parquet → Databricks Notebook: loads dataset (evidence: `README.md`)
- Databricks Notebook → Visual Analysis: produces visuals (evidence: `README.md`)
