# tmphan66/tfnsw-realtime-transit-tracker architecture

> Architecture diagram of the GitHub repository tmphan66/tfnsw-realtime-transit-tracker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tmphan66/tfnsw-realtime-transit-tracker
- Repository: https://github.com/tmphan66/tfnsw-realtime-transit-tracker
- Video tour (about a minute): https://gitdiagram.com/tmphan66/tfnsw-realtime-transit-tracker/video

## Overview

This is a real-time Sydney transit tracking and analytics platform. The main workflow polls TfNSW for vehicle and trip-update feeds, publishes them to Kafka, then uses Flink to decode and join updates, detect bunching, and write live state and historical events. dbt transforms historical events into analytics tables; the Streamlit dashboard combines live vehicle state, historical trends, and those tables for network overview and route/operator analysis. The excerpts show the main pipeline and dashboard reads; some dashboard and infrastructure wiring is unsampled.

Main entry points: [`dbt/models/staging/stg_vehicle_events.sql`](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/staging/stg_vehicle_events.sql) (Staging events), [`dashboard/app.py`](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py) (Streamlit dashboard), [`flink-job/flink_job.py`](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py) (Bunching detection), [`producer/producer.py`](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/producer/producer.py) (Feed poller)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Feed Ingestion"]
  node_producer["Feed poller<br/>[producer.py]"]
end

subgraph group_streaming["Stream Processing"]
  node_flink["Flink job<br/>[flink_job.py]"]
  node_decode["Feed decoding<br/>[flink_job.py]"]
  node_join["Delay attachment<br/>[flink_job.py]"]
  node_bunching["Bunching detection<br/>[flink_job.py]"]
end

subgraph group_analytics["Historical Analytics"]
  node_staging["Staging events"]
  node_busiest["Busiest routes<br/>[busiest_routes.sql]"]
  node_delay_hour["Delay by hour<br/>[delay_by_hour.sql]"]
  node_headway["Headway adherence"]
  node_operator_perf["Operator performance"]
  node_top_routes["Top delayed routes"]
end

subgraph group_dashboard["Dashboard"]
  node_dashboard_app["Streamlit dashboard<br/>[app.py]"]
  node_live_read["Live state lookup<br/>[app.py]"]
  node_history_read["Historical trend lookup<br/>[app.py]"]
  node_charts["Network and drill-down views<br/>[app.py]"]
end

node_rider(("Transit analyst"))
node_tfnsw["TfNSW feeds"]
node_kafka["Kafka topics"]
node_live_state[("Live vehicle state")]
node_silver[("Silver event files")]
node_gold[("Analytics tables")]

node_producer -->|"polls feeds"| node_tfnsw
node_producer -->|"publishes updates"| node_kafka
node_kafka -->|"supplies topics"| node_flink
node_flink -->|"maps records"| node_decode
node_decode -->|"feeds streams"| node_join
node_join -->|"enriched records"| node_bunching
node_bunching -->|"writes state"| node_live_state
node_bunching -->|"flushes Parquet"| node_silver
node_silver -->|"reads events"| node_staging
node_staging -->|"transforms"| node_busiest
node_staging -->|"transforms"| node_delay_hour
node_staging -->|"transforms"| node_headway
node_staging -->|"transforms"| node_operator_perf
node_staging -->|"transforms"| node_top_routes
node_busiest -->|"materializes"| node_gold
node_delay_hour -->|"materializes"| node_gold
node_headway -->|"materializes"| node_gold
node_operator_perf -->|"materializes"| node_gold
node_top_routes -->|"materializes"| node_gold
node_dashboard_app -->|"fetches vehicles"| node_live_read
node_dashboard_app -->|"fetches trend"| node_history_read
node_dashboard_app -->|"queries tables"| node_gold
node_live_read -->|"reads state"| node_live_state
node_history_read -->|"queries history"| node_silver
node_live_read -->|"supplies vehicles"| node_charts
node_history_read -->|"supplies trend"| node_charts
node_gold -->|"supplies metrics"| node_charts
node_rider -->|"views dashboard"| node_dashboard_app

click node_producer "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/producer/producer.py"
click node_flink "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py"
click node_decode "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py"
click node_join "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py"
click node_bunching "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py"
click node_staging "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/staging/stg_vehicle_events.sql"
click node_busiest "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/busiest_routes.sql"
click node_delay_hour "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/delay_by_hour.sql"
click node_headway "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/headway_adherence.sql"
click node_operator_perf "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/operator_performance.sql"
click node_top_routes "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/top_routes_by_delay.sql"
click node_dashboard_app "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py"
click node_live_read "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py"
click node_history_read "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py"
click node_charts "https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer toneBlue
class node_flink,node_decode,node_join,node_bunching,node_live_state,node_silver,node_gold toneAmber
class node_staging,node_busiest,node_delay_hour,node_headway,node_operator_perf,node_top_routes toneMint
class node_dashboard_app,node_live_read,node_history_read,node_charts toneRose
class node_rider,node_tfnsw,node_kafka toneIndigo
```

## Components

### Feed Ingestion

- [Feed poller](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/producer/producer.py): `producer/producer.py`

### Stream Processing

- [Flink job](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py): `flink-job/flink_job.py`
- [Feed decoding](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py): `flink-job/flink_job.py`
- [Delay attachment](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py): `flink-job/flink_job.py`
- [Bunching detection](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/flink-job/flink_job.py): `flink-job/flink_job.py`

### Historical Analytics

- [Staging events](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/staging/stg_vehicle_events.sql): `dbt/models/staging/stg_vehicle_events.sql`
- [Busiest routes](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/busiest_routes.sql): `dbt/models/marts/busiest_routes.sql`
- [Delay by hour](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/delay_by_hour.sql): `dbt/models/marts/delay_by_hour.sql`
- [Headway adherence](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/headway_adherence.sql): `dbt/models/marts/headway_adherence.sql`
- [Operator performance](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/operator_performance.sql): `dbt/models/marts/operator_performance.sql`
- [Top delayed routes](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dbt/models/marts/top_routes_by_delay.sql): `dbt/models/marts/top_routes_by_delay.sql`

### Dashboard

- [Streamlit dashboard](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py): `dashboard/app.py`
- [Live state lookup](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py): `dashboard/app.py`
- [Historical trend lookup](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py): `dashboard/app.py`
- [Network and drill-down views](https://github.com/tmphan66/tfnsw-realtime-transit-tracker/blob/main/dashboard/app.py): `dashboard/app.py`

### Other components

- **Transit analyst**
- **TfNSW feeds**
- **Kafka topics**
- **Live vehicle state**
- **Silver event files**
- **Analytics tables**

## Connections

- Feed poller → TfNSW feeds: polls feeds
- Feed poller → Kafka topics: publishes updates
- Kafka topics → Flink job: supplies topics
- Flink job → Feed decoding: maps records
- Feed decoding → Delay attachment: feeds streams
- Delay attachment → Bunching detection: enriched records
- Bunching detection → Live vehicle state: writes state
- Bunching detection → Silver event files: flushes Parquet
- Silver event files → Staging events: reads events
- Staging events → Busiest routes: transforms
- Staging events → Delay by hour: transforms
- Staging events → Headway adherence: transforms
- Staging events → Operator performance: transforms
- Staging events → Top delayed routes: transforms
- Busiest routes → Analytics tables: materializes
- Delay by hour → Analytics tables: materializes
- Headway adherence → Analytics tables: materializes
- Operator performance → Analytics tables: materializes
- Top delayed routes → Analytics tables: materializes
- Streamlit dashboard → Live state lookup: fetches vehicles
- Streamlit dashboard → Historical trend lookup: fetches trend
- Streamlit dashboard → Analytics tables: queries tables
- Live state lookup → Live vehicle state: reads state
- Historical trend lookup → Silver event files: queries history
- Live state lookup → Network and drill-down views: supplies vehicles
- Historical trend lookup → Network and drill-down views: supplies trend
- Analytics tables → Network and drill-down views: supplies metrics
- Transit analyst → Streamlit dashboard: views dashboard