How kathisnehith/nyc311-requests-etl-pipeline works
Project type and purpose
• It runs on GCP using Spark (DataProc), Airflow (Composer), Cloud Storage, BigQuery, and ends in Tableau dashboards.
File structure
• Top‐level Python scripts:
• NYC_datafetch_by_year.py & nyc_fetch_intial1M_batch.py → data‐ingestion/extract from the NYC 311 API
• Simple_dataproc_task_DAG.py → Airflow DAG definition to orchestrate DataProc jobs
• Notebook: nyc_datatransform_pysparkjob.ipynb → Spark transformations (clean, aggregate)
• SQL file: nyc311_requests_analysis.sql → query scripts against BigQuery
• README.md → mission, architecture, tech stack
README for additional insights
• Mentions Airflow on Composer orchestrating daily ETL
Essential elements
• Raw vs processed data zones
• Scheduling triggers and job dependencies
• Call out data formats (JSON/CSV) and update frequency (daily)