How tmphan66/tfnsw-realtime-transit-tracker works
This is a real-time Sydney transit tracking and analytics platform. The main workflow polls TfNSW for vehicle and trip-update feeds, publishes them to Kafka, then uses Flink to decode and join updates, detect bunching, and write live state and historical events. dbt transforms historical events into analytics tables; the Streamlit dashboard combines live vehicle state, historical trends, and those tables for network overview and route/operator analysis. The excerpts show the main pipeline and dashboard reads; some dashboard and infrastructure wiring is unsampled.
Main entry points: stg_vehicle_events.sql Staging events · app.py Streamlit dashboard · flink_job.py Bunching detection · producer.py Feed poller
Components 21
Feed Ingestion
- Feed poller
producer/producer.py
Stream Processing
- Flink job
flink-job/flink_job.py - Feed decoding
flink-job/flink_job.py - Delay attachment
flink-job/flink_job.py - Bunching detection
flink-job/flink_job.py
Historical Analytics
- Staging events
dbt/models/staging/stg_vehicle_events.sql - Busiest routes
dbt/models/marts/busiest_routes.sql - Delay by hour
dbt/models/marts/delay_by_hour.sql - Headway adherence
dbt/models/marts/headway_adherence.sql - Operator performance
dbt/models/marts/operator_performance.sql - Top delayed routes
dbt/models/marts/top_routes_by_delay.sql
Dashboard
- Streamlit dashboard
dashboard/app.py - Live state lookup
dashboard/app.py - Historical trend lookup
dashboard/app.py - Network and drill-down views
dashboard/app.py
Other components
- Transit analyst
- TfNSW feeds
- Kafka topics
- Live vehicle state
- Silver event files
- Analytics tables
Connections 28
- Feed poller → TfNSW feeds: polls feeds
- Feed poller → Kafka topics: publishes updates
- Kafka topics → Flink job: supplies topics
- Flink job → Feed decoding: maps records
- Feed decoding → Delay attachment: feeds streams
- Delay attachment → Bunching detection: enriched records
- Bunching detection → Live vehicle state: writes state
- Bunching detection → Silver event files: flushes Parquet
- Silver event files → Staging events: reads events
- Staging events → Busiest routes: transforms
- Staging events → Delay by hour: transforms
- Staging events → Headway adherence: transforms
- Staging events → Operator performance: transforms
- Staging events → Top delayed routes: transforms
- Busiest routes → Analytics tables: materializes
- Delay by hour → Analytics tables: materializes
- Headway adherence → Analytics tables: materializes
- Operator performance → Analytics tables: materializes
- Top delayed routes → Analytics tables: materializes
- Streamlit dashboard → Live state lookup: fetches vehicles
- Streamlit dashboard → Historical trend lookup: fetches trend
- Streamlit dashboard → Analytics tables: queries tables
- Live state lookup → Live vehicle state: reads state
- Historical trend lookup → Silver event files: queries history
- Live state lookup → Network and drill-down views: supplies vehicles
- Historical trend lookup → Network and drill-down views: supplies trend
- Analytics tables → Network and drill-down views: supplies metrics
- Transit analyst → Streamlit dashboard: views dashboard