How ciskander2/citi-bike-kafka-pipeline works
The repository implements a Citi Bike streaming analytics pipeline: GBFS station updates are published to Kafka, consumed in batches into S3 and PostgreSQL, optionally transformed into Parquet with Spark, modeled with dbt, and queried by a Streamlit dashboard. The main streaming and dashboard paths are supported by excerpts; dbt behavior and several pipeline links are documented in the README, while model internals were not sampled.
Main entry points: app.py Streamlit dashboard · staging Staging models · simple_consumer.py Batched consumer · citibike_producer.py Kafka producer
Components 15
Event ingestion
- Kafka producer
src/citibike_producer.py - Station status topic
- Batched consumer
src/simple_consumer.py
Data storage
- Raw JSONL lake
- Postgres warehouse
- Parquet outputs
Analytics transforms
- Spark transform
src/spark_transformation.py - Staging models
citibike_dbt/models/staging - Top stations
citibike_dbt/models/staging/top_20_stations.sql - No docking analysis
citibike_dbt/models/staging/no_docking_but_availability.sql - Capacity analysis
citibike_dbt/models/staging/capacity_zero_availability.sql - Hourly status
citibike_dbt/models/staging/hourly_bike_status.sql - Utilization ratio
citibike_dbt/models/staging/utlilization_ratio.sql
Analytics dashboard
- Streamlit dashboard
dashboard/app.py
Other components
- GBFS feed
Connections 19
- GBFS feed → Kafka producer: provides updates
README.md - Kafka producer → Station status topic: publishes events
src/citibike_producer.py - Station status topic → Batched consumer: delivers events
README.md - Batched consumer → Raw JSONL lake: writes batches
src/simple_consumer.py - Batched consumer → Postgres warehouse: inserts batches
src/simple_consumer.py - Raw JSONL lake → Spark transform: reads JSONL
src/spark_transformation.py - Spark transform → Parquet outputs: writes Parquet
src/spark_transformation.py - Postgres warehouse → Staging models: models warehouse data
README.md - Staging models → Top stations: includes
- Staging models → No docking analysis: includes
- Staging models → Capacity analysis: includes
- Staging models → Hourly status: includes
- Staging models → Utilization ratio: includes
- Streamlit dashboard → Postgres warehouse: queries metrics
dashboard/app.py - Streamlit dashboard → Top stations: queries model
dashboard/app.py - Streamlit dashboard → No docking analysis: queries model
dashboard/app.py - Streamlit dashboard → Capacity analysis: queries model
dashboard/app.py - Streamlit dashboard → Hourly status: queries model
dashboard/app.py - Streamlit dashboard → Utilization ratio: queries model
dashboard/app.py