How jacobwongers/nhl-game-stat-pipeline works
This project presents skater goals, assists, and shots from a past NHL game alongside that skater’s season averages. Its actual-stats workflow fetches and flattens game events, replays them to Kafka, and relies on the documented Databricks/S3 processing path to supply the actual view. The separate expected-stats workflow extracts team season records and loads them through GCS into BigQuery for the expected view. The tree and README document Databricks jobs and the Streamlit interface, but their source is not included; those nodes use their enclosing directories.
Main entry points: extract_batch.py Extract team batch · fetch_game.py Fetch game · load_to_bigquery.py Load BigQuery · app Streamlit views
Components 21
Actual game feed
- Fetch game
pipelines/actual/ingestion/fetch_game.py - Flatten plays
pipelines/actual/ingestion/extract_plays.py - Raw game JSON
data/raw/nhl - Extracted events
data/extracted - Replay events
pipelines/actual/producer/producer.py - Databricks processing
pipelines/actual/databricks
Expected stats
- Fetch team stats
pipelines/expected/batch/hockey_reference/fetch_expected_stats.py - Extract team batch
pipelines/expected/batch/hockey_reference/extract_batch.py - Expected JSONL
data/expected - Calculate averages
pipelines/expected/batch/calculate_expected_stats.py - Upload to GCS
pipelines/expected/batch/upload_to_gcs.py - Load BigQuery
pipelines/expected/batch/load_to_bigquery.py
Storage and display
- Streamlit views
app
Other components
- Viewer
- NHL API
- Kafka
- S3
- Delta tables
- Hockey Reference
- Google Cloud Storage
- BigQuery
Connections 21
- Fetch game → NHL API: requests game
- Fetch game → Raw game JSON: writes JSON
- Fetch game → Flatten plays: calls extraction
- Flatten plays → Extracted events: writes JSONL
- Replay events → Extracted events: reads events
- Replay events → Kafka: publishes events
- Kafka → S3: delivers events
- Databricks processing → S3: reads events
- Databricks processing → Delta tables: writes tables
- Delta tables → Streamlit views: supplies actual stats
- Extract team batch → Fetch team stats: calls fetcher
- Fetch team stats → Hockey Reference: fetches stats
- Extract team batch → Expected JSONL: writes JSONL
- Extract team batch → Upload to GCS: uploads file
- Upload to GCS → Google Cloud Storage: stores object
- Extract team batch → Load BigQuery: loads URI
- Load BigQuery → Google Cloud Storage: reads object
- Load BigQuery → BigQuery: loads rows
- BigQuery → Streamlit views: supplies averages
- Viewer → Streamlit views: selects skater
- Calculate averages → Expected JSONL: reads or writes stats