How jan-ric/dep-data-engineering-jan-ric works
This repository implements Phase 1 of a solar-inverter analysis workflow: fetch hourly weather data, derive plant and inverter performance metrics, and save datasets for analysis. The main calculation expands weather-derived features across 830 inverters, applies test-only fault offsets, and calculates performance ratio. The README also describes anomaly identification as the project goal. The dashboard and SQL files are present in the tree, but their behavior is not shown in the available excerpts, so their connections are omitted.
Main entry points: transform.py Validation and Profiling · ingest.py Fetch Weather · raw Raw Weather JSON · processed Cleaned Weather CSV
Components 14
Weather Ingestion
- Fetch Weather
scripts/ingest.py - Raw Weather JSON
data/raw
Metric Engineering
- Load Weather
scripts/transform.py - Plant Sizing
scripts/transform.py - Temperature Models
scripts/transform.py - Power Models
scripts/transform.py - Inverter Fault Scenarios
scripts/transform.py - Performance Ratio
scripts/transform.py - Validation and Profiling
scripts/transform.py
Analysis Data
- Cleaned Weather CSV
data/processed - Computation CSV
data/processed - Inverter Metrics CSV
data/processed
Other components
- Analyst
- Open-Meteo
Connections 16
- Analyst → Fetch Weather: runs
scripts/ingest.py - Fetch Weather → Open-Meteo: requests weather
scripts/ingest.py - Fetch Weather → Raw Weather JSON: writes JSON
scripts/ingest.py - Raw Weather JSON → Load Weather: provides input
scripts/transform.py - Load Weather → Plant Sizing: supplies weather
scripts/transform.py - Load Weather → Temperature Models: supplies weather
scripts/transform.py - Plant Sizing → Power Models: provides capacity
scripts/transform.py - Temperature Models → Power Models: provides temperature
scripts/transform.py - Power Models → Performance Ratio: provides output
scripts/transform.py - Temperature Models → Inverter Fault Scenarios: provides baseline
scripts/transform.py - Inverter Fault Scenarios → Performance Ratio: provides inverter output
scripts/transform.py - Performance Ratio → Validation and Profiling: checks metrics
scripts/transform.py - Validation and Profiling → Cleaned Weather CSV: writes weather
scripts/transform.py - Validation and Profiling → Computation CSV: writes metrics
scripts/transform.py - Validation and Profiling → Inverter Metrics CSV: writes query data
scripts/transform.py - Analyst → Validation and Profiling: reviews results