How ankurchavda/streamify works
Streamify generates simulated music-service events, streams them through Kafka and Spark into a GCS data lake, then uses Airflow to stage hourly event data in BigQuery and run dbt transformations for analytics. A separate Airflow workflow loads song reference data. Terraform provisions cloud infrastructure, and the documented dashboard uses the resulting analytics. Some subsystems are represented from the README and tree rather than sampled implementation; their links are limited to documented behavior.
Main entry points: streamify_dag.py Hourly event DAG · stream_all_events.py Spark stream app · load_songs_dag.py Song reference load · task_templates.py BigQuery task operators
Components 18
Event generation
- Eventsim
eventsim
Streaming ingestion
- Kafka topics
kafka - Spark stream app
spark_streaming/stream_all_events.py - Stream processing
spark_streaming/streaming_functions.py - Event schemas
spark_streaming/schema.py
Batch orchestration
- Hourly event DAG
airflow/dags/streamify_dag.py - BigQuery task operators
airflow/dags/task_templates.py - Staging schemas
airflow/dags/schema.py - Event staging SQL
airflow/dags/sql - Song reference load
airflow/dags/load_songs_dag.py
Analytics modeling
- Song seed data
dbt/seeds/songs.csv - dbt project
dbt - Dimensions and facts
dbt/models/core
Cloud platform
- GCS data lake
- BigQuery warehouse
- Cloud provisioning
terraform/main.tf - Google Cloud
Other components
- Analytics dashboard
Connections 18
- Eventsim → Kafka topics: publishes events
- Spark stream app → Kafka topics: reads topics
- Spark stream app → Stream processing: calls processing
- Spark stream app → Event schemas: uses schemas
- Stream processing → GCS data lake: writes Parquet
- Hourly event DAG → BigQuery task operators: creates tasks
- Hourly event DAG → Staging schemas: loads schemas
- Hourly event DAG → Event staging SQL: loads queries
- BigQuery task operators → GCS data lake: reads event files
- BigQuery task operators → BigQuery warehouse: stages events
- Hourly event DAG → dbt project: runs dbt
- dbt project → BigQuery warehouse: reads and builds models
- dbt project → Dimensions and facts: executes models
- Song reference load → Song seed data: loads reference data
- Song reference load → GCS data lake: uploads Parquet
- Song reference load → BigQuery warehouse: creates external table
- Dimensions and facts → Analytics dashboard: provides analytics
- Cloud provisioning → Google Cloud: provisions infrastructure