# ankurchavda/streamify architecture

> Architecture diagram of the GitHub repository ankurchavda/streamify, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ankurchavda/streamify
- Repository: https://github.com/ankurchavda/streamify

## Overview

Streamify generates simulated music-service events, streams them through Kafka and Spark into a GCS data lake, then uses Airflow to stage hourly event data in BigQuery and run dbt transformations for analytics. A separate Airflow workflow loads song reference data. Terraform provisions cloud infrastructure, and the documented dashboard uses the resulting analytics. Some subsystems are represented from the README and tree rather than sampled implementation; their links are limited to documented behavior.

Main entry points: [`airflow/dags/streamify_dag.py`](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/streamify_dag.py) (Hourly event DAG), [`spark_streaming/stream_all_events.py`](https://github.com/ankurchavda/streamify/blob/main/spark_streaming/stream_all_events.py) (Spark stream app), [`airflow/dags/load_songs_dag.py`](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/load_songs_dag.py) (Song reference load), [`airflow/dags/task_templates.py`](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/task_templates.py) (BigQuery task operators)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Event generation"]
  node_simulator["Eventsim"]
end

subgraph group_streaming["Streaming ingestion"]
  node_kafka["Kafka topics"]
  node_spark_app["Spark stream app"]
  node_spark_helpers["Stream processing"]
  node_spark_schema["Event schemas<br/>[schema.py]"]
end

subgraph group_orchestration["Batch orchestration"]
  node_airflow_main["Hourly event DAG<br/>[streamify_dag.py]"]
  node_bq_tasks["BigQuery task operators<br/>[task_templates.py]"]
  node_bq_schema["Staging schemas<br/>[schema.py]"]
  node_event_sql["Event staging SQL"]
  node_songs_dag["Song reference load<br/>[load_songs_dag.py]"]
end

subgraph group_analytics["Analytics modeling"]
  node_songs_seed["Song seed data<br/>[songs.csv]"]
  node_dbt_project["dbt project"]
  node_core_models["Dimensions and facts"]
end

subgraph group_platform["Cloud platform"]
  node_gcs[("GCS data lake")]
  node_bigquery[("BigQuery warehouse")]
  node_terraform["Cloud provisioning<br/>[main.tf]"]
  node_gcp[("Google Cloud")]
end

node_dashboard(("Analytics dashboard"))

node_simulator -->|"publishes events"| node_kafka
node_spark_app -->|"reads topics"| node_kafka
node_spark_app -->|"calls processing"| node_spark_helpers
node_spark_app -->|"uses schemas"| node_spark_schema
node_spark_helpers -->|"writes Parquet"| node_gcs
node_airflow_main -->|"creates tasks"| node_bq_tasks
node_airflow_main -->|"loads schemas"| node_bq_schema
node_airflow_main -->|"loads queries"| node_event_sql
node_bq_tasks -->|"reads event files"| node_gcs
node_bq_tasks -->|"stages events"| node_bigquery
node_airflow_main -->|"runs dbt"| node_dbt_project
node_dbt_project -->|"reads and builds models"| node_bigquery
node_dbt_project -->|"executes models"| node_core_models
node_songs_dag -->|"loads reference data"| node_songs_seed
node_songs_dag -->|"uploads Parquet"| node_gcs
node_songs_dag -->|"creates external table"| node_bigquery
node_core_models -.->|"provides analytics"| node_dashboard
node_terraform -->|"provisions infrastructure"| node_gcp

click node_simulator "https://github.com/ankurchavda/streamify/tree/main/eventsim"
click node_kafka "https://github.com/ankurchavda/streamify/tree/main/kafka"
click node_spark_app "https://github.com/ankurchavda/streamify/blob/main/spark_streaming/stream_all_events.py"
click node_spark_helpers "https://github.com/ankurchavda/streamify/blob/main/spark_streaming/streaming_functions.py"
click node_spark_schema "https://github.com/ankurchavda/streamify/blob/main/spark_streaming/schema.py"
click node_airflow_main "https://github.com/ankurchavda/streamify/blob/main/airflow/dags/streamify_dag.py"
click node_bq_tasks "https://github.com/ankurchavda/streamify/blob/main/airflow/dags/task_templates.py"
click node_bq_schema "https://github.com/ankurchavda/streamify/blob/main/airflow/dags/schema.py"
click node_event_sql "https://github.com/ankurchavda/streamify/tree/main/airflow/dags/sql"
click node_songs_dag "https://github.com/ankurchavda/streamify/blob/main/airflow/dags/load_songs_dag.py"
click node_songs_seed "https://github.com/ankurchavda/streamify/blob/main/dbt/seeds/songs.csv"
click node_dbt_project "https://github.com/ankurchavda/streamify/tree/main/dbt"
click node_core_models "https://github.com/ankurchavda/streamify/tree/main/dbt/models/core"
click node_terraform "https://github.com/ankurchavda/streamify/blob/main/terraform/main.tf"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simulator toneBlue
class node_kafka,node_spark_app,node_spark_helpers,node_spark_schema toneAmber
class node_airflow_main,node_bq_tasks,node_bq_schema,node_event_sql,node_songs_dag toneMint
class node_songs_seed,node_dbt_project,node_core_models toneRose
class node_gcs,node_bigquery,node_terraform,node_gcp,node_dashboard toneIndigo
```

## Components

### Event generation

- [Eventsim](https://github.com/ankurchavda/streamify/tree/main/eventsim): `eventsim`

### Streaming ingestion

- [Kafka topics](https://github.com/ankurchavda/streamify/tree/main/kafka): `kafka`
- [Spark stream app](https://github.com/ankurchavda/streamify/blob/main/spark_streaming/stream_all_events.py): `spark_streaming/stream_all_events.py`
- [Stream processing](https://github.com/ankurchavda/streamify/blob/main/spark_streaming/streaming_functions.py): `spark_streaming/streaming_functions.py`
- [Event schemas](https://github.com/ankurchavda/streamify/blob/main/spark_streaming/schema.py): `spark_streaming/schema.py`

### Batch orchestration

- [Hourly event DAG](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/streamify_dag.py): `airflow/dags/streamify_dag.py`
- [BigQuery task operators](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/task_templates.py): `airflow/dags/task_templates.py`
- [Staging schemas](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/schema.py): `airflow/dags/schema.py`
- [Event staging SQL](https://github.com/ankurchavda/streamify/tree/main/airflow/dags/sql): `airflow/dags/sql`
- [Song reference load](https://github.com/ankurchavda/streamify/blob/main/airflow/dags/load_songs_dag.py): `airflow/dags/load_songs_dag.py`

### Analytics modeling

- [Song seed data](https://github.com/ankurchavda/streamify/blob/main/dbt/seeds/songs.csv): `dbt/seeds/songs.csv`
- [dbt project](https://github.com/ankurchavda/streamify/tree/main/dbt): `dbt`
- [Dimensions and facts](https://github.com/ankurchavda/streamify/tree/main/dbt/models/core): `dbt/models/core`

### Cloud platform

- **GCS data lake**
- **BigQuery warehouse**
- [Cloud provisioning](https://github.com/ankurchavda/streamify/blob/main/terraform/main.tf): `terraform/main.tf`
- **Google Cloud**

### Other components

- **Analytics dashboard**

## Connections

- Eventsim → Kafka topics: publishes events
- Spark stream app → Kafka topics: reads topics
- Spark stream app → Stream processing: calls processing
- Spark stream app → Event schemas: uses schemas
- Stream processing → GCS data lake: writes Parquet
- Hourly event DAG → BigQuery task operators: creates tasks
- Hourly event DAG → Staging schemas: loads schemas
- Hourly event DAG → Event staging SQL: loads queries
- BigQuery task operators → GCS data lake: reads event files
- BigQuery task operators → BigQuery warehouse: stages events
- Hourly event DAG → dbt project: runs dbt
- dbt project → BigQuery warehouse: reads and builds models
- dbt project → Dimensions and facts: executes models
- Song reference load → Song seed data: loads reference data
- Song reference load → GCS data lake: uploads Parquet
- Song reference load → BigQuery warehouse: creates external table
- Dimensions and facts → Analytics dashboard: provides analytics
- Cloud provisioning → Google Cloud: provisions infrastructure
