# terk93/plugspot architecture

> Architecture diagram of the GitHub repository terk93/plugspot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/terk93/plugspot
- Repository: https://github.com/terk93/plugspot

## Overview

plugspot is a Databricks data warehouse that supports decisions about where to place 50 new Swedish EV fast-charging stations. Its principal workflow fetches SCB, Trafikanalys and electricity-price data into a landing volume, loads bronze tables, transforms and tests silver and gold data with dbt, then exposes municipality and electricity-price marts for analysis. The Databricks job coordinates ingestion, dbt build and read-only validation. The graph follows the README and sampled ingestion code; dbt model-level dependencies were not sampled, so the transformation stages are shown at layer level.

Main entry points: [`src/ingest/run.py`](https://github.com/terk93/plugspot/blob/main/src/ingest/run.py) (Ingestion runner), [`src/tools/readonly_checks.py`](https://github.com/terk93/plugspot/blob/main/src/tools/readonly_checks.py) (Read-only validation), [`src/ingest/sources/scb.py`](https://github.com/terk93/plugspot/blob/main/src/ingest/sources/scb.py) (SCB extraction), [`src/ingest/sources/trafikanalys.py`](https://github.com/terk93/plugspot/blob/main/src/ingest/sources/trafikanalys.py) (Vehicle extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["External sources"]
  node_scb_api["SCB PxWebApi"]
  node_trafikanalys_api["Trafikanalys API"]
  node_elpriset_api["Elpriset API"]
end

subgraph group_ingestion["Ingestion"]
  node_pipeline_job["Pipeline job"]
  node_ingest_run["Ingestion runner<br/>[run.py]"]
  node_scb_ingest["SCB extraction<br/>[scb.py]"]
  node_trafikanalys_ingest["Vehicle extraction<br/>[trafikanalys.py]"]
  node_price_ingest["Price extraction<br/>[elpriset.py]"]
  node_http_client["HTTP client<br/>[http_client.py]"]
  node_landing_store["Landing storage<br/>[landing.py]"]
  node_bronze_loader["Bronze loader<br/>[bronze.py]"]
end

subgraph group_warehouse["Warehouse"]
  node_landing_volume[("Landing volume")]
  node_bronze_tables[("Bronze tables")]
  node_ops_log[("Run log<br/>[03_ops_tables.sql]")]
  node_silver_models["Silver models"]
  node_dq_quarantine[("DQ quarantine")]
  node_gold_models["Gold star schema"]
end

subgraph group_delivery["Delivery"]
  node_serving_marts["Serving marts"]
  node_validation["Read-only validation<br/>[readonly_checks.py]"]
end

node_decision_user(("Decision analyst"))

node_pipeline_job -->|"dispatches ingestion"| node_ingest_run
node_pipeline_job -->|"runs dbt build"| node_silver_models
node_pipeline_job -->|"dispatches checks"| node_validation
node_scb_api -->|"provides data"| node_scb_ingest
node_trafikanalys_api -->|"provides data"| node_trafikanalys_ingest
node_elpriset_api -->|"provides data"| node_price_ingest
node_ingest_run -->|"dispatches extraction"| node_scb_ingest
node_ingest_run -->|"dispatches extraction"| node_trafikanalys_ingest
node_ingest_run -->|"dispatches extraction"| node_price_ingest
node_scb_ingest -->|"fetches metadata and CSV"| node_http_client
node_trafikanalys_ingest -->|"fetches data"| node_http_client
node_price_ingest -->|"fetches prices"| node_http_client
node_scb_ingest -->|"writes snapshots"| node_landing_store
node_trafikanalys_ingest -->|"writes snapshots"| node_landing_store
node_price_ingest -->|"writes price files"| node_landing_store
node_landing_store -->|"stores raw files"| node_landing_volume
node_ingest_run -->|"loads landed files"| node_bronze_loader
node_bronze_loader -->|"reads files"| node_landing_volume
node_bronze_loader -->|"COPY INTO"| node_bronze_tables
node_ingest_run -->|"records run status"| node_ops_log
node_bronze_tables -->|"transforms source rows"| node_silver_models
node_silver_models -->|"routes failing rows"| node_dq_quarantine
node_silver_models -->|"builds dimensions and facts"| node_gold_models
node_gold_models -->|"builds decision marts"| node_serving_marts
node_decision_user -->|"queries results"| node_serving_marts
node_validation -->|"checks load invariants"| node_bronze_tables
node_validation -->|"checks row counts"| node_gold_models
node_validation -->|"checks municipality coverage"| node_serving_marts

click node_pipeline_job "https://github.com/terk93/plugspot/blob/main/resources/jobs/plugspot_pipeline.yml"
click node_ingest_run "https://github.com/terk93/plugspot/blob/main/src/ingest/run.py"
click node_scb_ingest "https://github.com/terk93/plugspot/blob/main/src/ingest/sources/scb.py"
click node_trafikanalys_ingest "https://github.com/terk93/plugspot/blob/main/src/ingest/sources/trafikanalys.py"
click node_price_ingest "https://github.com/terk93/plugspot/blob/main/src/ingest/sources/elpriset.py"
click node_http_client "https://github.com/terk93/plugspot/blob/main/src/ingest/common/http_client.py"
click node_landing_store "https://github.com/terk93/plugspot/blob/main/src/ingest/common/landing.py"
click node_bronze_loader "https://github.com/terk93/plugspot/blob/main/src/ingest/common/bronze.py"
click node_ops_log "https://github.com/terk93/plugspot/blob/main/src/setup/03_ops_tables.sql"
click node_silver_models "https://github.com/terk93/plugspot/tree/main/dbt/models/silver"
click node_dq_quarantine "https://github.com/terk93/plugspot/tree/main/dbt/models/silver/quarantine"
click node_gold_models "https://github.com/terk93/plugspot/tree/main/dbt/models/gold"
click node_serving_marts "https://github.com/terk93/plugspot/tree/main/dbt/models/serving"
click node_validation "https://github.com/terk93/plugspot/blob/main/src/tools/readonly_checks.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scb_api,node_trafikanalys_api,node_elpriset_api toneBlue
class node_pipeline_job,node_ingest_run,node_scb_ingest,node_trafikanalys_ingest,node_price_ingest,node_http_client,node_landing_store,node_bronze_loader toneAmber
class node_landing_volume,node_bronze_tables,node_ops_log,node_silver_models,node_dq_quarantine,node_gold_models toneMint
class node_serving_marts,node_validation toneRose
class node_decision_user toneIndigo
```

## Components

### External sources

- **SCB PxWebApi**
- **Trafikanalys API**
- **Elpriset API**

### Ingestion

- [Pipeline job](https://github.com/terk93/plugspot/blob/main/resources/jobs/plugspot_pipeline.yml): `resources/jobs/plugspot_pipeline.yml`
- [Ingestion runner](https://github.com/terk93/plugspot/blob/main/src/ingest/run.py): `src/ingest/run.py`
- [SCB extraction](https://github.com/terk93/plugspot/blob/main/src/ingest/sources/scb.py): `src/ingest/sources/scb.py`
- [Vehicle extraction](https://github.com/terk93/plugspot/blob/main/src/ingest/sources/trafikanalys.py): `src/ingest/sources/trafikanalys.py`
- [Price extraction](https://github.com/terk93/plugspot/blob/main/src/ingest/sources/elpriset.py): `src/ingest/sources/elpriset.py`
- [HTTP client](https://github.com/terk93/plugspot/blob/main/src/ingest/common/http_client.py): `src/ingest/common/http_client.py`
- [Landing storage](https://github.com/terk93/plugspot/blob/main/src/ingest/common/landing.py): `src/ingest/common/landing.py`
- [Bronze loader](https://github.com/terk93/plugspot/blob/main/src/ingest/common/bronze.py): `src/ingest/common/bronze.py`

### Warehouse

- **Landing volume**
- **Bronze tables**
- [Run log](https://github.com/terk93/plugspot/blob/main/src/setup/03_ops_tables.sql): `src/setup/03_ops_tables.sql`
- [Silver models](https://github.com/terk93/plugspot/tree/main/dbt/models/silver): `dbt/models/silver`
- [DQ quarantine](https://github.com/terk93/plugspot/tree/main/dbt/models/silver/quarantine): `dbt/models/silver/quarantine`
- [Gold star schema](https://github.com/terk93/plugspot/tree/main/dbt/models/gold): `dbt/models/gold`

### Delivery

- [Serving marts](https://github.com/terk93/plugspot/tree/main/dbt/models/serving): `dbt/models/serving`
- [Read-only validation](https://github.com/terk93/plugspot/blob/main/src/tools/readonly_checks.py): `src/tools/readonly_checks.py`

### Other components

- **Decision analyst**

## Connections

- Pipeline job → Ingestion runner: dispatches ingestion
- Pipeline job → Silver models: runs dbt build
- Pipeline job → Read-only validation: dispatches checks
- SCB PxWebApi → SCB extraction: provides data
- Trafikanalys API → Vehicle extraction: provides data
- Elpriset API → Price extraction: provides data
- Ingestion runner → SCB extraction: dispatches extraction
- Ingestion runner → Vehicle extraction: dispatches extraction
- Ingestion runner → Price extraction: dispatches extraction
- SCB extraction → HTTP client: fetches metadata and CSV
- Vehicle extraction → HTTP client: fetches data
- Price extraction → HTTP client: fetches prices
- SCB extraction → Landing storage: writes snapshots
- Vehicle extraction → Landing storage: writes snapshots
- Price extraction → Landing storage: writes price files
- Landing storage → Landing volume: stores raw files
- Ingestion runner → Bronze loader: loads landed files
- Bronze loader → Landing volume: reads files
- Bronze loader → Bronze tables: COPY INTO
- Ingestion runner → Run log: records run status
- Bronze tables → Silver models: transforms source rows
- Silver models → DQ quarantine: routes failing rows
- Silver models → Gold star schema: builds dimensions and facts
- Gold star schema → Serving marts: builds decision marts
- Decision analyst → Serving marts: queries results
- Read-only validation → Bronze tables: checks load invariants
- Read-only validation → Gold star schema: checks row counts
- Read-only validation → Serving marts: checks municipality coverage