# ciskander2/ev_charging_infrastructure_pipeline architecture

> Architecture diagram of the GitHub repository ciskander2/ev_charging_infrastructure_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ciskander2/ev_charging_infrastructure_pipeline
- Repository: https://github.com/ciskander2/ev_charging_infrastructure_pipeline

## Overview

This repository is a batch ETL pipeline for turning NREL EV charging-station records into a PostgreSQL/Supabase analytics dataset. The principal workflow extracts paginated API results into S3, normalizes and enriches them into Parquet, then loads the processed data into the warehouse; Airflow runs the three steps in sequence. SQL supports warehouse setup, infrastructure analysis, and data-quality checks. The source excerpts show the script-level transfers and DAG dependencies; SQL-file relationships are based on README descriptions, not sampled SQL contents.

Main entry points: [`dags/ev_file_airflow.py`](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/dags/ev_file_airflow.py) (Airflow DAG), [`scripts/transform_to_parquet.py`](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/transform_to_parquet.py) (Data Transform), [`scripts/upload_to_supabase.py`](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/upload_to_supabase.py) (Warehouse Loader), [`scripts/extract_api_to_s3.py`](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/extract_api_to_s3.py) (API Extractor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG<br/>[ev_file_airflow.py]"]
end

subgraph group_pipeline["ETL Pipeline"]
  node_extract["API Extractor"]
  node_transform["Data Transform"]
  node_normalize["Normalize and Enrich"]
  node_load["Warehouse Loader"]
end

subgraph group_storage["Data Stores"]
  node_raw[("Raw S3 JSON")]
  node_processed[("Processed S3 Parquet")]
  node_warehouse[("PostgreSQL Warehouse")]
end

subgraph group_analysis["Warehouse Analysis"]
  node_schema["Warehouse Schema"]
  node_analytics["SQL Analytics"]
  node_quality["Data Quality Checks"]
end

node_operator(("Pipeline Operator"))
node_nrel["NREL API"]
node_analyst(("Data Analyst"))

node_operator -->|"triggers DAG"| node_airflow
node_airflow -->|"runs first"| node_extract
node_airflow -->|"runs second"| node_transform
node_airflow -->|"runs third"| node_load
node_nrel -->|"serves station data"| node_extract
node_extract -->|"writes JSON pages"| node_raw
node_transform -->|"reads JSON pages"| node_raw
node_transform -->|"normalizes records"| node_normalize
node_normalize -->|"writes Parquet"| node_processed
node_load -->|"reads Parquet"| node_processed
node_load -->|"loads table"| node_warehouse
node_analyst -->|"runs queries"| node_analytics
node_analytics -->|"analyzes data"| node_warehouse
node_analyst -->|"runs checks"| node_quality
node_quality -->|"validates data"| node_warehouse
node_schema -->|"defines table"| node_warehouse

click node_airflow "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/dags/ev_file_airflow.py"
click node_extract "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/extract_api_to_s3.py"
click node_transform "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/transform_to_parquet.py"
click node_normalize "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/transform_to_parquet.py"
click node_load "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/upload_to_supabase.py"
click node_schema "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/01_create_ev_stations_table.sql"
click node_analytics "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/02_analytics_queries.sql"
click node_quality "https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/03_data_quality_checks.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow toneBlue
class node_extract,node_transform,node_normalize,node_load toneAmber
class node_raw,node_processed,node_warehouse,node_nrel toneMint
class node_schema,node_analytics,node_quality toneRose
class node_operator,node_analyst toneIndigo
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/dags/ev_file_airflow.py): `dags/ev_file_airflow.py`

### ETL Pipeline

- [API Extractor](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/extract_api_to_s3.py): `scripts/extract_api_to_s3.py`
- [Data Transform](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/transform_to_parquet.py): `scripts/transform_to_parquet.py`
- [Normalize and Enrich](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/transform_to_parquet.py): `scripts/transform_to_parquet.py`
- [Warehouse Loader](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/scripts/upload_to_supabase.py): `scripts/upload_to_supabase.py`

### Data Stores

- **Raw S3 JSON**
- **Processed S3 Parquet**
- **PostgreSQL Warehouse**

### Warehouse Analysis

- [Warehouse Schema](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/01_create_ev_stations_table.sql): `sql/01_create_ev_stations_table.sql`
- [SQL Analytics](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/02_analytics_queries.sql): `sql/02_analytics_queries.sql`
- [Data Quality Checks](https://github.com/ciskander2/ev_charging_infrastructure_pipeline/blob/main/sql/03_data_quality_checks.sql): `sql/03_data_quality_checks.sql`

### Other components

- **Pipeline Operator**
- **NREL API**
- **Data Analyst**

## Connections

- Pipeline Operator → Airflow DAG: triggers DAG (evidence: `README.md`)
- Airflow DAG → API Extractor: runs first (evidence: `dags/ev_file_airflow.py`)
- Airflow DAG → Data Transform: runs second (evidence: `dags/ev_file_airflow.py`)
- Airflow DAG → Warehouse Loader: runs third (evidence: `dags/ev_file_airflow.py`)
- NREL API → API Extractor: serves station data (evidence: `README.md`)
- API Extractor → Raw S3 JSON: writes JSON pages (evidence: `scripts/extract_api_to_s3.py`)
- Data Transform → Raw S3 JSON: reads JSON pages (evidence: `scripts/transform_to_parquet.py`)
- Data Transform → Normalize and Enrich: normalizes records (evidence: `scripts/transform_to_parquet.py`)
- Normalize and Enrich → Processed S3 Parquet: writes Parquet (evidence: `scripts/transform_to_parquet.py`)
- Warehouse Loader → Processed S3 Parquet: reads Parquet (evidence: `scripts/upload_to_supabase.py`)
- Warehouse Loader → PostgreSQL Warehouse: loads table (evidence: `scripts/upload_to_supabase.py`)
- Data Analyst → SQL Analytics: runs queries (evidence: `README.md`)
- SQL Analytics → PostgreSQL Warehouse: analyzes data (evidence: `README.md`)
- Data Analyst → Data Quality Checks: runs checks (evidence: `README.md`)
- Data Quality Checks → PostgreSQL Warehouse: validates data (evidence: `README.md`)
- Warehouse Schema → PostgreSQL Warehouse: defines table (evidence: `README.md`)
