How ciskander2/ev_charging_infrastructure_pipeline works
This repository is a batch ETL pipeline for turning NREL EV charging-station records into a PostgreSQL/Supabase analytics dataset. The principal workflow extracts paginated API results into S3, normalizes and enriches them into Parquet, then loads the processed data into the warehouse; Airflow runs the three steps in sequence. SQL supports warehouse setup, infrastructure analysis, and data-quality checks. The source excerpts show the script-level transfers and DAG dependencies; SQL-file relationships are based on README descriptions, not sampled SQL contents.
Main entry points: ev_file_airflow.py Airflow DAG · transform_to_parquet.py Data Transform · upload_to_supabase.py Warehouse Loader · extract_api_to_s3.py API Extractor
Components 14
Orchestration
- Airflow DAG
dags/ev_file_airflow.py
ETL Pipeline
- API Extractor
scripts/extract_api_to_s3.py - Data Transform
scripts/transform_to_parquet.py - Normalize and Enrich
scripts/transform_to_parquet.py - Warehouse Loader
scripts/upload_to_supabase.py
Data Stores
- Raw S3 JSON
- Processed S3 Parquet
- PostgreSQL Warehouse
Warehouse Analysis
- Warehouse Schema
sql/01_create_ev_stations_table.sql - SQL Analytics
sql/02_analytics_queries.sql - Data Quality Checks
sql/03_data_quality_checks.sql
Other components
- Pipeline Operator
- NREL API
- Data Analyst
Connections 16
- Pipeline Operator → Airflow DAG: triggers DAG
README.md - Airflow DAG → API Extractor: runs first
dags/ev_file_airflow.py - Airflow DAG → Data Transform: runs second
dags/ev_file_airflow.py - Airflow DAG → Warehouse Loader: runs third
dags/ev_file_airflow.py - NREL API → API Extractor: serves station data
README.md - API Extractor → Raw S3 JSON: writes JSON pages
scripts/extract_api_to_s3.py - Data Transform → Raw S3 JSON: reads JSON pages
scripts/transform_to_parquet.py - Data Transform → Normalize and Enrich: normalizes records
scripts/transform_to_parquet.py - Normalize and Enrich → Processed S3 Parquet: writes Parquet
scripts/transform_to_parquet.py - Warehouse Loader → Processed S3 Parquet: reads Parquet
scripts/upload_to_supabase.py - Warehouse Loader → PostgreSQL Warehouse: loads table
scripts/upload_to_supabase.py - Data Analyst → SQL Analytics: runs queries
README.md - SQL Analytics → PostgreSQL Warehouse: analyzes data
README.md - Data Analyst → Data Quality Checks: runs checks
README.md - Data Quality Checks → PostgreSQL Warehouse: validates data
README.md - Warehouse Schema → PostgreSQL Warehouse: defines table
README.md