# kathisnehith/nyc311-requests-etl-pipeline architecture

> Architecture diagram of the GitHub repository kathisnehith/nyc311-requests-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kathisnehith/nyc311-requests-etl-pipeline
- Repository: https://github.com/kathisnehith/nyc311-requests-etl-pipeline

## Overview

##### Project type and purpose
- It runs on GCP using Spark (DataProc), Airflow (Composer), Cloud Storage, BigQuery, and ends in Tableau dashboards.

##### File structure
- Top‐level Python scripts:
- NYC_datafetch_by_year.py & nyc_fetch_intial1M_batch.py → data‐ingestion/extract from the NYC 311 API
- Simple_dataproc_task_DAG.py → Airflow DAG definition to orchestrate DataProc jobs
- Notebook: nyc_datatransform_pysparkjob.ipynb → Spark transformations (clean, aggregate)
- SQL file: nyc311_requests_analysis.sql → query scripts against BigQuery
- README.md → mission, architecture, tech stack

##### README for additional insights
- Mentions Airflow on Composer orchestrating daily ETL

##### Essential elements
- Raw vs processed data zones
- Scheduling triggers and job dependencies
- Call out data formats (JSON/CSV) and update frequency (daily)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    API["NYC 311 REST API"]:::external

    %% Orchestration Layer
    Airflow["Airflow (Composer)"]:::orch

    %% Compute Layer
    subgraph "Compute: DataProc Cluster"
        FetchYear["Extract Job: NYC_datafetch_by_year.py"]:::compute
        FetchBatch["Extract Job: nyc_fetch_intial1M_batch.py"]:::compute
        Transform["Transform Job: nyc_datatransform_pysparkjob.ipynb"]:::compute
    end

    %% Storage Layer
    subgraph "Storage: GCP"
        GCSRaw["GCS Raw Bucket"]:::storage
        GCSProcessed["GCS Processed Bucket"]:::storage
        BigQuery["BigQuery: 311_Tables"]:::storage
    end

    %% Analysis & Visualization
    SQLNode["SQL Analysis: nyc311_requests_analysis.sql"]:::analysis
    Tableau["Tableau Dashboards"]:::viz

    %% Data Flow
    API -->|ingest calls| Airflow
    Airflow -->|triggers| FetchYear
    Airflow -->|triggers| FetchBatch
    FetchYear -->|writes JSON/CSV| GCSRaw
    FetchBatch -->|writes JSON/CSV| GCSRaw
    GCSRaw -->|reads raw data| Transform
    Transform -->|writes processed data| GCSProcessed
    Transform -->|loads final tables| BigQuery
    BigQuery -->|queried by| SQLNode
    BigQuery -->|visualized by| Tableau

    %% Click Events
    click FetchYear "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/NYC_datafetch_by_year.py"
    click FetchBatch "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/nyc_fetch_intial1M_batch.py"
    click Airflow "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/Simple_dataproc_task_DAG.py"
    click Transform "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/nyc_datatransform_pysparkjob.ipynb"
    click SQLNode "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/nyc311_requests_analysis.sql"
    click README "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/blob/main/README.md"
    click Img "https://github.com/kathisnehith/nyc311-requests-etl-pipeline/tree/main/img/sample"

    %% Styles
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef orch fill:#ffa500,stroke:#333,stroke-width:1px
    classDef compute fill:#90ee90,stroke:#333,stroke-width:1px
    classDef storage fill:#add8e6,stroke:#333,stroke-width:1px
    classDef analysis fill:#add8e6,stroke:#333,stroke-width:1px
    classDef viz fill:#dda0dd,stroke:#333,stroke-width:1px
```
