# kenzacha/energy-charts-data-engineering-challenge architecture

> Architecture diagram of the GitHub repository kenzacha/energy-charts-data-engineering-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kenzacha/energy-charts-data-engineering-challenge
- Repository: https://github.com/kenzacha/energy-charts-data-engineering-challenge

## Overview

This local PySpark pipeline fetches daily production and price data from Energy-Charts, stores raw JSON in Bronze Delta tables, normalizes it into Silver event rows, and computes Gold daily aggregates. The user-configured entry point calls the pipeline runner, which owns the day-by-day ingestion and subsequent transformations. Spark configuration is an internal runtime layer. The source excerpt shows the stage implementations and calls; no separate orchestration or analytics service is present.

Main entry points: [`src/pipeline.py`](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py) (Pipeline runner), [`main.py`](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/main.py) (Run entry point)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Execution"]
  node_main["Run entry point<br/>[main.py]"]
  node_runner["Pipeline runner<br/>[pipeline.py]"]
  node_spark["Spark setup<br/>[pipeline.py]"]
end

subgraph group_bronze["Bronze ingestion"]
  node_power_ingest["Power ingestion<br/>[pipeline.py]"]
  node_price_ingest["Price ingestion<br/>[pipeline.py]"]
  node_bronze_power[("Raw power table<br/>[pipeline.py]")]
  node_bronze_price[("Raw price table<br/>[pipeline.py]")]
end

subgraph group_silver["Silver events"]
  node_power_transform["Power normalization<br/>[pipeline.py]"]
  node_price_transform["Price normalization<br/>[pipeline.py]"]
  node_silver_power[("Power events<br/>[pipeline.py]")]
  node_silver_price[("Price events<br/>[pipeline.py]")]
end

subgraph group_gold["Gold analytics"]
  node_gold_builder["Daily aggregation<br/>[pipeline.py]"]
  node_gold_power[("Daily power<br/>[pipeline.py]")]
  node_gold_price[("Daily price<br/>[pipeline.py]")]
end

node_user(("Pipeline operator"))
node_energy_api["Energy-Charts API"]

node_user -->|"starts"| node_main
node_main -->|"calls"| node_runner
node_runner -->|"creates"| node_spark
node_runner -->|"calls"| node_power_ingest
node_runner -->|"calls"| node_price_ingest
node_power_ingest -->|"fetches power"| node_energy_api
node_price_ingest -->|"fetches prices"| node_energy_api
node_power_ingest -->|"writes raw"| node_bronze_power
node_price_ingest -->|"writes raw"| node_bronze_price
node_runner -->|"calls"| node_power_transform
node_runner -->|"calls"| node_price_transform
node_power_transform -->|"reads raw"| node_bronze_power
node_price_transform -->|"reads raw"| node_bronze_price
node_power_transform -->|"writes events"| node_silver_power
node_price_transform -->|"writes events"| node_silver_price
node_runner -->|"calls"| node_gold_builder
node_gold_builder -->|"reads events"| node_silver_power
node_gold_builder -->|"reads events"| node_silver_price
node_gold_builder -->|"writes totals"| node_gold_power
node_gold_builder -->|"writes averages"| node_gold_price

click node_main "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/main.py"
click node_runner "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_spark "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_power_ingest "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_price_ingest "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_bronze_power "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_bronze_price "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_power_transform "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_price_transform "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_silver_power "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_silver_price "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_gold_builder "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_gold_power "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"
click node_gold_price "https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_runner,node_spark toneBlue
class node_power_ingest,node_price_ingest,node_bronze_power,node_bronze_price toneAmber
class node_power_transform,node_price_transform,node_silver_power,node_silver_price,node_energy_api toneMint
class node_gold_builder,node_gold_power,node_gold_price toneRose
class node_user toneIndigo
```

## Components

### Execution

- [Run entry point](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/main.py): `main.py`
- [Pipeline runner](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Spark setup](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`

### Bronze ingestion

- [Power ingestion](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Price ingestion](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Raw power table](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Raw price table](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`

### Silver events

- [Power normalization](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Price normalization](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Power events](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Price events](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`

### Gold analytics

- [Daily aggregation](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Daily power](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`
- [Daily price](https://github.com/kenzacha/energy-charts-data-engineering-challenge/blob/main/src/pipeline.py): `src/pipeline.py`

### Other components

- **Pipeline operator**
- **Energy-Charts API**

## Connections

- Pipeline operator → Run entry point: starts (evidence: `README.md`)
- Run entry point → Pipeline runner: calls (evidence: `main.py`)
- Pipeline runner → Spark setup: creates (evidence: `src/pipeline.py`)
- Pipeline runner → Power ingestion: calls (evidence: `src/pipeline.py`)
- Pipeline runner → Price ingestion: calls (evidence: `src/pipeline.py`)
- Power ingestion → Energy-Charts API: fetches power (evidence: `src/pipeline.py`)
- Price ingestion → Energy-Charts API: fetches prices (evidence: `src/pipeline.py`)
- Power ingestion → Raw power table: writes raw (evidence: `src/pipeline.py`)
- Price ingestion → Raw price table: writes raw (evidence: `src/pipeline.py`)
- Pipeline runner → Power normalization: calls (evidence: `src/pipeline.py`)
- Pipeline runner → Price normalization: calls (evidence: `src/pipeline.py`)
- Power normalization → Raw power table: reads raw (evidence: `src/pipeline.py`)
- Price normalization → Raw price table: reads raw (evidence: `src/pipeline.py`)
- Power normalization → Power events: writes events (evidence: `src/pipeline.py`)
- Price normalization → Price events: writes events (evidence: `src/pipeline.py`)
- Pipeline runner → Daily aggregation: calls (evidence: `src/pipeline.py`)
- Daily aggregation → Power events: reads events (evidence: `src/pipeline.py`)
- Daily aggregation → Price events: reads events (evidence: `src/pipeline.py`)
- Daily aggregation → Daily power: writes totals (evidence: `src/pipeline.py`)
- Daily aggregation → Daily price: writes averages (evidence: `src/pipeline.py`)
