# jacobwongers/nhl-game-stat-pipeline architecture

> Architecture diagram of the GitHub repository jacobwongers/nhl-game-stat-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jacobwongers/nhl-game-stat-pipeline
- Repository: https://github.com/jacobwongers/nhl-game-stat-pipeline

## Overview

This project presents skater goals, assists, and shots from a past NHL game alongside that skater’s season averages. Its actual-stats workflow fetches and flattens game events, replays them to Kafka, and relies on the documented Databricks/S3 processing path to supply the actual view. The separate expected-stats workflow extracts team season records and loads them through GCS into BigQuery for the expected view. The tree and README document Databricks jobs and the Streamlit interface, but their source is not included; those nodes use their enclosing directories.

Main entry points: [`pipelines/expected/batch/hockey_reference/extract_batch.py`](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/hockey_reference/extract_batch.py) (Extract team batch), [`pipelines/actual/ingestion/fetch_game.py`](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/ingestion/fetch_game.py) (Fetch game), [`pipelines/expected/batch/load_to_bigquery.py`](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/load_to_bigquery.py) (Load BigQuery), [`app`](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/app) (Streamlit views)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_actual["Actual game feed"]
  node_fetch["Fetch game<br/>[fetch_game.py]"]
  node_flatten["Flatten plays<br/>[extract_plays.py]"]
  node_rawgame["Raw game JSON"]
  node_eventfile["Extracted events"]
  node_replay["Replay events<br/>[producer.py]"]
  node_databricks["Databricks processing"]
end

subgraph group_expected["Expected stats"]
  node_fetchstats["Fetch team stats"]
  node_extractbatch["Extract team batch<br/>[extract_batch.py]"]
  node_expectedfile["Expected JSONL"]
  node_calculate["Calculate averages"]
  node_gcsupload["Upload to GCS<br/>[upload_to_gcs.py]"]
  node_bigqueryload["Load BigQuery"]
end

subgraph group_serving["Storage and display"]
  node_ui["Streamlit views"]
end

node_viewer(("Viewer"))
node_nhl{{"NHL API"}}
node_kafka["Kafka"]
node_s3[("S3")]
node_delta[("Delta tables")]
node_hockeyref{{"Hockey Reference"}}
node_gcs[("Google Cloud Storage")]
node_bigquery[("BigQuery")]

node_fetch -->|"requests game"| node_nhl
node_fetch -->|"writes JSON"| node_rawgame
node_fetch -->|"calls extraction"| node_flatten
node_flatten -->|"writes JSONL"| node_eventfile
node_replay -->|"reads events"| node_eventfile
node_replay -->|"publishes events"| node_kafka
node_kafka -->|"delivers events"| node_s3
node_databricks -->|"reads events"| node_s3
node_databricks -->|"writes tables"| node_delta
node_delta -->|"supplies actual stats"| node_ui
node_extractbatch -->|"calls fetcher"| node_fetchstats
node_fetchstats -->|"fetches stats"| node_hockeyref
node_extractbatch -->|"writes JSONL"| node_expectedfile
node_extractbatch -->|"uploads file"| node_gcsupload
node_gcsupload -->|"stores object"| node_gcs
node_extractbatch -->|"loads URI"| node_bigqueryload
node_bigqueryload -->|"reads object"| node_gcs
node_bigqueryload -->|"loads rows"| node_bigquery
node_bigquery -->|"supplies averages"| node_ui
node_viewer -->|"selects skater"| node_ui
node_calculate -.->|"reads or writes stats"| node_expectedfile

click node_fetch "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/ingestion/fetch_game.py"
click node_flatten "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/ingestion/extract_plays.py"
click node_rawgame "https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/raw/nhl"
click node_eventfile "https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/extracted"
click node_replay "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/producer/producer.py"
click node_databricks "https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/pipelines/actual/databricks"
click node_fetchstats "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/hockey_reference/fetch_expected_stats.py"
click node_extractbatch "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/hockey_reference/extract_batch.py"
click node_expectedfile "https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/expected"
click node_calculate "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/calculate_expected_stats.py"
click node_gcsupload "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/upload_to_gcs.py"
click node_bigqueryload "https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/load_to_bigquery.py"
click node_ui "https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/app"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetch,node_flatten,node_rawgame,node_eventfile,node_replay,node_databricks,node_viewer toneBlue
class node_fetchstats,node_extractbatch,node_expectedfile,node_calculate,node_gcsupload,node_bigqueryload,node_s3,node_delta,node_gcs,node_bigquery toneAmber
class node_ui,node_nhl toneMint
class node_kafka,node_hockeyref toneIndigo
```

## Components

### Actual game feed

- [Fetch game](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/ingestion/fetch_game.py): `pipelines/actual/ingestion/fetch_game.py`
- [Flatten plays](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/ingestion/extract_plays.py): `pipelines/actual/ingestion/extract_plays.py`
- [Raw game JSON](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/raw/nhl): `data/raw/nhl`
- [Extracted events](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/extracted): `data/extracted`
- [Replay events](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/actual/producer/producer.py): `pipelines/actual/producer/producer.py`
- [Databricks processing](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/pipelines/actual/databricks): `pipelines/actual/databricks`

### Expected stats

- [Fetch team stats](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/hockey_reference/fetch_expected_stats.py): `pipelines/expected/batch/hockey_reference/fetch_expected_stats.py`
- [Extract team batch](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/hockey_reference/extract_batch.py): `pipelines/expected/batch/hockey_reference/extract_batch.py`
- [Expected JSONL](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/data/expected): `data/expected`
- [Calculate averages](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/calculate_expected_stats.py): `pipelines/expected/batch/calculate_expected_stats.py`
- [Upload to GCS](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/upload_to_gcs.py): `pipelines/expected/batch/upload_to_gcs.py`
- [Load BigQuery](https://github.com/jacobwongers/nhl-game-stat-pipeline/blob/main/pipelines/expected/batch/load_to_bigquery.py): `pipelines/expected/batch/load_to_bigquery.py`

### Storage and display

- [Streamlit views](https://github.com/jacobwongers/nhl-game-stat-pipeline/tree/main/app): `app`

### Other components

- **Viewer**
- **NHL API**
- **Kafka**
- **S3**
- **Delta tables**
- **Hockey Reference**
- **Google Cloud Storage**
- **BigQuery**

## Connections

- Fetch game → NHL API: requests game
- Fetch game → Raw game JSON: writes JSON
- Fetch game → Flatten plays: calls extraction
- Flatten plays → Extracted events: writes JSONL
- Replay events → Extracted events: reads events
- Replay events → Kafka: publishes events
- Kafka → S3: delivers events
- Databricks processing → S3: reads events
- Databricks processing → Delta tables: writes tables
- Delta tables → Streamlit views: supplies actual stats
- Extract team batch → Fetch team stats: calls fetcher
- Fetch team stats → Hockey Reference: fetches stats
- Extract team batch → Expected JSONL: writes JSONL
- Extract team batch → Upload to GCS: uploads file
- Upload to GCS → Google Cloud Storage: stores object
- Extract team batch → Load BigQuery: loads URI
- Load BigQuery → Google Cloud Storage: reads object
- Load BigQuery → BigQuery: loads rows
- BigQuery → Streamlit views: supplies averages
- Viewer → Streamlit views: selects skater
- Calculate averages → Expected JSONL: reads or writes stats