How terk93/plugspot works
plugspot is a Databricks data warehouse that supports decisions about where to place 50 new Swedish EV fast-charging stations. Its principal workflow fetches SCB, Trafikanalys and electricity-price data into a landing volume, loads bronze tables, transforms and tests silver and gold data with dbt, then exposes municipality and electricity-price marts for analysis. The Databricks job coordinates ingestion, dbt build and read-only validation. The graph follows the README and sampled ingestion code; dbt model-level dependencies were not sampled, so the transformation stages are shown at layer level.
Main entry points: run.py Ingestion runner · readonly_checks.py Read-only validation · scb.py SCB extraction · trafikanalys.py Vehicle extraction
Components 20
External sources
- SCB PxWebApi
- Trafikanalys API
- Elpriset API
Ingestion
- Pipeline job
resources/jobs/plugspot_pipeline.yml - Ingestion runner
src/ingest/run.py - SCB extraction
src/ingest/sources/scb.py - Vehicle extraction
src/ingest/sources/trafikanalys.py - Price extraction
src/ingest/sources/elpriset.py - HTTP client
src/ingest/common/http_client.py - Landing storage
src/ingest/common/landing.py - Bronze loader
src/ingest/common/bronze.py
Warehouse
- Landing volume
- Bronze tables
- Run log
src/setup/03_ops_tables.sql - Silver models
dbt/models/silver - DQ quarantine
dbt/models/silver/quarantine - Gold star schema
dbt/models/gold
Delivery
- Serving marts
dbt/models/serving - Read-only validation
src/tools/readonly_checks.py
Other components
- Decision analyst
Connections 28
- Pipeline job → Ingestion runner: dispatches ingestion
- Pipeline job → Silver models: runs dbt build
- Pipeline job → Read-only validation: dispatches checks
- SCB PxWebApi → SCB extraction: provides data
- Trafikanalys API → Vehicle extraction: provides data
- Elpriset API → Price extraction: provides data
- Ingestion runner → SCB extraction: dispatches extraction
- Ingestion runner → Vehicle extraction: dispatches extraction
- Ingestion runner → Price extraction: dispatches extraction
- SCB extraction → HTTP client: fetches metadata and CSV
- Vehicle extraction → HTTP client: fetches data
- Price extraction → HTTP client: fetches prices
- SCB extraction → Landing storage: writes snapshots
- Vehicle extraction → Landing storage: writes snapshots
- Price extraction → Landing storage: writes price files
- Landing storage → Landing volume: stores raw files
- Ingestion runner → Bronze loader: loads landed files
- Bronze loader → Landing volume: reads files
- Bronze loader → Bronze tables: COPY INTO
- Ingestion runner → Run log: records run status
- Bronze tables → Silver models: transforms source rows
- Silver models → DQ quarantine: routes failing rows
- Silver models → Gold star schema: builds dimensions and facts
- Gold star schema → Serving marts: builds decision marts
- Decision analyst → Serving marts: queries results
- Read-only validation → Bronze tables: checks load invariants
- Read-only validation → Gold star schema: checks row counts
- Read-only validation → Serving marts: checks municipality coverage