How sakho115/f1_azure works
This is a local-first Formula 1 analytics lakehouse that ingests historical and API-sourced data, processes it through Bronze, Silver, and Gold tiers, validates the resulting datasets, and presents analytics in a Streamlit dashboard. The documented main workflow is driven by the Python orchestrator; Silver transformations and merges produce the inputs to Gold marts. The graph also shows dashboard views and governance as product capabilities. Some connections are documented rather than directly visible in sampled code, and the API integration is not shown wired into the main ingestion path.
Main entry points: pipeline.py Pipeline Orchestrator · app.py Streamlit Dashboard · data Silver Tables · merge_manager.py Silver Merge Manager
Components 19
Pipeline Orchestration
- Pipeline Orchestrator
orchestration/pipeline.py - Spark Session
src/utils/spark.py
Lakehouse Data
- Historical Inputs
data/sample - Bronze Ingestion
src/ingestion/file_ingestion.py - Bronze Tables
data - Silver Transformations
src/transformations - Silver Merge Manager
src/silver/merge_manager.py - Silver Tables
data
Analytics and Quality
- Standings Marts
src/gold/standings.py - Telemetry Marts
src/gold/telemetry.py - Gold Tables
data - Data Quality
src/quality/expectations.py - SQL Governance
sql/governance.sql
Dashboard Serving
- Streamlit Dashboard
dashboard/app.py - Standings View
dashboard/components/standings_view.py - Telemetry View
dashboard/components/telemetry_view.py - Circuit View
dashboard/components/circuit_view.py
Other components
- Pipeline Operator
- Ergast API
Connections 24
- Pipeline Operator → Pipeline Orchestrator: starts
README.md - Pipeline Orchestrator → Spark Session: initializes
orchestration/pipeline.py - Pipeline Orchestrator → Bronze Ingestion: runs Bronze
orchestration/pipeline.py - Bronze Ingestion → Historical Inputs: reads fallback
src/ingestion/file_ingestion.py - Bronze Ingestion → Bronze Tables: writes Delta
src/ingestion/file_ingestion.py - Ergast API → Bronze Ingestion: may supply data
- Pipeline Orchestrator → Silver Merge Manager: runs Silver
orchestration/pipeline.py - Silver Merge Manager → Bronze Tables: reads Delta
src/silver/merge_manager.py - Silver Merge Manager → Silver Transformations: applies transforms
src/silver/merge_manager.py - Silver Merge Manager → Silver Tables: merges tables
src/silver/merge_manager.py - Pipeline Orchestrator → Standings Marts: builds standings
orchestration/pipeline.py - Pipeline Orchestrator → Telemetry Marts: builds telemetry
orchestration/pipeline.py - Standings Marts → Silver Tables: reads tables
src/gold/standings.py - Telemetry Marts → Silver Tables: reads tables
src/gold/telemetry.py - Standings Marts → Gold Tables: writes marts
src/gold/standings.py - Telemetry Marts → Gold Tables: writes marts
src/gold/telemetry.py - Pipeline Orchestrator → Data Quality: runs validation
orchestration/pipeline.py - Data Quality → Silver Tables: checks tables
src/quality/expectations.py - Data Quality → Gold Tables: checks tables
src/quality/expectations.py - Streamlit Dashboard → Gold Tables: loads analytics
dashboard/app.py - Streamlit Dashboard → Silver Tables: loads circuits
dashboard/app.py - Streamlit Dashboard → Standings View: renders
dashboard/app.py - Streamlit Dashboard → Telemetry View: renders
dashboard/app.py - Streamlit Dashboard → Circuit View: renders
dashboard/app.py