# jan-ric/dep-data-engineering-jan-ric architecture

> Architecture diagram of the GitHub repository jan-ric/dep-data-engineering-jan-ric, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jan-ric/dep-data-engineering-jan-ric
- Repository: https://github.com/jan-ric/dep-data-engineering-jan-ric

## Overview

This repository implements Phase 1 of a solar-inverter analysis workflow: fetch hourly weather data, derive plant and inverter performance metrics, and save datasets for analysis. The main calculation expands weather-derived features across 830 inverters, applies test-only fault offsets, and calculates performance ratio. The README also describes anomaly identification as the project goal. The dashboard and SQL files are present in the tree, but their behavior is not shown in the available excerpts, so their connections are omitted.

Main entry points: [`scripts/transform.py`](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py) (Validation and Profiling), [`scripts/ingest.py`](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/ingest.py) (Fetch Weather), [`data/raw`](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/raw) (Raw Weather JSON), [`data/processed`](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed) (Cleaned Weather CSV)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Weather Ingestion"]
  node_fetch["Fetch Weather<br/>[ingest.py]"]
  node_raw[("Raw Weather JSON")]
end

subgraph group_engineering["Metric Engineering"]
  node_load["Load Weather<br/>[transform.py]"]
  node_sizing["Plant Sizing<br/>[transform.py]"]
  node_temperature["Temperature Models<br/>[transform.py]"]
  node_power["Power Models<br/>[transform.py]"]
  node_faults["Inverter Fault Scenarios<br/>[transform.py]"]
  node_pr["Performance Ratio<br/>[transform.py]"]
  node_validation["Validation and Profiling<br/>[transform.py]"]
end

subgraph group_outputs["Analysis Data"]
  node_cleaned[("Cleaned Weather CSV")]
  node_computation[("Computation CSV")]
  node_querydata[("Inverter Metrics CSV")]
end

node_operator(("Analyst"))
node_openmeteo["Open-Meteo"]

node_operator -->|"runs"| node_fetch
node_fetch -->|"requests weather"| node_openmeteo
node_fetch -->|"writes JSON"| node_raw
node_raw -->|"provides input"| node_load
node_load -->|"supplies weather"| node_sizing
node_load -->|"supplies weather"| node_temperature
node_sizing -->|"provides capacity"| node_power
node_temperature -->|"provides temperature"| node_power
node_power -->|"provides output"| node_pr
node_temperature -->|"provides baseline"| node_faults
node_faults -->|"provides inverter output"| node_pr
node_pr -->|"checks metrics"| node_validation
node_validation -->|"writes weather"| node_cleaned
node_validation -->|"writes metrics"| node_computation
node_validation -->|"writes query data"| node_querydata
node_operator -.->|"reviews results"| node_validation

click node_fetch "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/ingest.py"
click node_raw "https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/raw"
click node_load "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_sizing "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_temperature "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_power "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_faults "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_pr "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_validation "https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py"
click node_cleaned "https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed"
click node_computation "https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed"
click node_querydata "https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetch,node_raw toneBlue
class node_load,node_sizing,node_temperature,node_power,node_faults,node_pr,node_validation toneAmber
class node_cleaned,node_computation,node_querydata toneMint
class node_operator,node_openmeteo toneIndigo
```

## Components

### Weather Ingestion

- [Fetch Weather](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/ingest.py): `scripts/ingest.py`
- [Raw Weather JSON](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/raw): `data/raw`

### Metric Engineering

- [Load Weather](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Plant Sizing](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Temperature Models](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Power Models](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Inverter Fault Scenarios](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Performance Ratio](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`
- [Validation and Profiling](https://github.com/jan-ric/dep-data-engineering-jan-ric/blob/main/scripts/transform.py): `scripts/transform.py`

### Analysis Data

- [Cleaned Weather CSV](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed): `data/processed`
- [Computation CSV](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed): `data/processed`
- [Inverter Metrics CSV](https://github.com/jan-ric/dep-data-engineering-jan-ric/tree/main/data/processed): `data/processed`

### Other components

- **Analyst**
- **Open-Meteo**

## Connections

- Analyst → Fetch Weather: runs (evidence: `scripts/ingest.py`)
- Fetch Weather → Open-Meteo: requests weather (evidence: `scripts/ingest.py`)
- Fetch Weather → Raw Weather JSON: writes JSON (evidence: `scripts/ingest.py`)
- Raw Weather JSON → Load Weather: provides input (evidence: `scripts/transform.py`)
- Load Weather → Plant Sizing: supplies weather (evidence: `scripts/transform.py`)
- Load Weather → Temperature Models: supplies weather (evidence: `scripts/transform.py`)
- Plant Sizing → Power Models: provides capacity (evidence: `scripts/transform.py`)
- Temperature Models → Power Models: provides temperature (evidence: `scripts/transform.py`)
- Power Models → Performance Ratio: provides output (evidence: `scripts/transform.py`)
- Temperature Models → Inverter Fault Scenarios: provides baseline (evidence: `scripts/transform.py`)
- Inverter Fault Scenarios → Performance Ratio: provides inverter output (evidence: `scripts/transform.py`)
- Performance Ratio → Validation and Profiling: checks metrics (evidence: `scripts/transform.py`)
- Validation and Profiling → Cleaned Weather CSV: writes weather (evidence: `scripts/transform.py`)
- Validation and Profiling → Computation CSV: writes metrics (evidence: `scripts/transform.py`)
- Validation and Profiling → Inverter Metrics CSV: writes query data (evidence: `scripts/transform.py`)
- Analyst → Validation and Profiling: reviews results