# dvpombo/solete architecture

> Architecture diagram of the GitHub repository dvpombo/solete, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dvpombo/solete
- Repository: https://github.com/dvpombo/solete

## Overview

SOLETE combines a version-4 dataset cleaning and quality-control pipeline with a time-series forecasting platform and reproducible benchmarks. The data-preparation path cleans raw measurements, resamples and exports them; the forecasting path loads SOLETE data, prepares features, trains models and evaluates predictions. The README documents the benchmark and data-quality/diagnostic capabilities as additional branches. Some subsystem links rely on the SOURCE INDEX or README because their source was not excerpted; uncertain implementation details are omitted.

Main entry points: [`dataset/pipeline/clean_solete_1sec.py`](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/clean_solete_1sec.py) (1-sec cleaning), [`scripts/quickstart/MLForecasting.py`](https://github.com/dvpombo/solete/blob/main/scripts/quickstart/MLForecasting.py) (Forecast quickstart), [`solete/io.py`](https://github.com/dvpombo/solete/blob/main/solete/io.py) (Data loading and expansion), [`solete/preprocessing.py`](https://github.com/dvpombo/solete/blob/main/solete/preprocessing.py) (Feature preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_pipeline["Dataset pipeline"]
  node_cleaner["1-sec cleaning"]
  node_qc_flags["Cleaning QC rules<br/>[qc_flags.py]"]
  node_solar_position["Solar position<br/>[solar_position.py]"]
  node_resampler["Resolution resampling<br/>[resample_solete.py]"]
  node_parquet_export["Parquet export<br/>[export_parquet.py]"]
  node_reports["Pipeline reports<br/>[solete_report.py]"]
  node_diagnostics["Data diagnostics"]
end

subgraph group_forecast["Forecast platform"]
  node_forecast_script["Forecast quickstart<br/>[MLForecasting.py]"]
  node_data_io["Data loading and expansion<br/>[io.py]"]
  node_preprocessing["Feature preparation<br/>[preprocessing.py]"]
  node_modeling["Forecast models<br/>[modeling.py]"]
  node_metrics["Forecast metrics<br/>[metrics.py]"]
  node_postprocess["Forecast postprocessing<br/>[postprocess.py]"]
end

subgraph group_shared["Shared utilities"]
  node_paths["Data path resolution<br/>[paths.py]"]
  node_platform_qc["Platform QC<br/>[qc.py]"]
  node_physics["PV physics<br/>[physics.py]"]
end

subgraph group_analysis["Analysis and benchmarks"]
  node_inspection["Dataset inspection<br/>[inspect_dataset.py]"]
  node_benchmarks["Forecast benchmarks"]
end

node_researcher(("Researcher"))
node_figshare[("Figshare dataset")]
node_source_data[("Raw and cleaned data")]

node_researcher -->|"downloads data"| node_figshare
node_figshare -->|"provides files"| node_source_data
node_researcher -->|"runs cleaning"| node_cleaner
node_cleaner -->|"uses QC rules"| node_qc_flags
node_cleaner -->|"computes solar position"| node_solar_position
node_cleaner -->|"prints report"| node_reports
node_cleaner -->|"reads and writes"| node_source_data
node_researcher -->|"runs resampling"| node_resampler
node_resampler -->|"reads and writes"| node_source_data
node_resampler -->|"uses report helper"| node_reports
node_researcher -->|"runs export"| node_parquet_export
node_parquet_export -->|"exports files"| node_source_data
node_researcher -->|"investigates quality"| node_diagnostics
node_researcher -->|"configures forecasts"| node_forecast_script
node_forecast_script -->|"loads data"| node_data_io
node_forecast_script -->|"prepares features"| node_preprocessing
node_forecast_script -->|"trains models"| node_modeling
node_forecast_script -->|"processes predictions"| node_postprocess
node_data_io -->|"resolves files"| node_paths
node_data_io -->|"expands data"| node_preprocessing
node_data_io -->|"applies QC"| node_platform_qc
node_data_io -->|"uses helpers"| node_postprocess
node_preprocessing -->|"uses physics features"| node_physics
node_preprocessing -->|"uses QC precedence"| node_platform_qc
node_postprocess -->|"writes outputs"| node_paths
node_researcher -->|"inspects dataset"| node_inspection
node_inspection -->|"resolves data files"| node_paths
node_researcher -->|"reproduces benchmarks"| node_benchmarks

click node_source_data "https://github.com/dvpombo/solete/tree/main/data/hdf5"
click node_cleaner "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/clean_solete_1sec.py"
click node_qc_flags "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/qc_flags.py"
click node_solar_position "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/solar_position.py"
click node_resampler "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/resample_solete.py"
click node_parquet_export "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/export_parquet.py"
click node_reports "https://github.com/dvpombo/solete/blob/main/dataset/pipeline/solete_report.py"
click node_diagnostics "https://github.com/dvpombo/solete/tree/main/dataset/diagnostics"
click node_forecast_script "https://github.com/dvpombo/solete/blob/main/scripts/quickstart/MLForecasting.py"
click node_data_io "https://github.com/dvpombo/solete/blob/main/solete/io.py"
click node_preprocessing "https://github.com/dvpombo/solete/blob/main/solete/preprocessing.py"
click node_modeling "https://github.com/dvpombo/solete/blob/main/solete/modeling.py"
click node_metrics "https://github.com/dvpombo/solete/blob/main/solete/metrics.py"
click node_postprocess "https://github.com/dvpombo/solete/blob/main/solete/postprocess.py"
click node_paths "https://github.com/dvpombo/solete/blob/main/solete/paths.py"
click node_platform_qc "https://github.com/dvpombo/solete/blob/main/solete/qc.py"
click node_physics "https://github.com/dvpombo/solete/blob/main/solete/physics.py"
click node_inspection "https://github.com/dvpombo/solete/blob/main/scripts/inspect_dataset.py"
click node_benchmarks "https://github.com/dvpombo/solete/tree/main/benchmarks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cleaner,node_qc_flags,node_solar_position,node_resampler,node_parquet_export,node_reports,node_diagnostics toneBlue
class node_forecast_script,node_data_io,node_preprocessing,node_modeling,node_metrics,node_postprocess,node_figshare,node_source_data toneAmber
class node_paths,node_platform_qc,node_physics toneMint
class node_inspection,node_benchmarks toneRose
class node_researcher toneIndigo
```

## Components

### Dataset pipeline

- [1-sec cleaning](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/clean_solete_1sec.py): `dataset/pipeline/clean_solete_1sec.py`
- [Cleaning QC rules](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/qc_flags.py): `dataset/pipeline/qc_flags.py`
- [Solar position](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/solar_position.py): `dataset/pipeline/solar_position.py`
- [Resolution resampling](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/resample_solete.py): `dataset/pipeline/resample_solete.py`
- [Parquet export](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/export_parquet.py): `dataset/pipeline/export_parquet.py`
- [Pipeline reports](https://github.com/dvpombo/solete/blob/main/dataset/pipeline/solete_report.py): `dataset/pipeline/solete_report.py`
- [Data diagnostics](https://github.com/dvpombo/solete/tree/main/dataset/diagnostics): `dataset/diagnostics`

### Forecast platform

- [Forecast quickstart](https://github.com/dvpombo/solete/blob/main/scripts/quickstart/MLForecasting.py): `scripts/quickstart/MLForecasting.py`
- [Data loading and expansion](https://github.com/dvpombo/solete/blob/main/solete/io.py): `solete/io.py`
- [Feature preparation](https://github.com/dvpombo/solete/blob/main/solete/preprocessing.py): `solete/preprocessing.py`
- [Forecast models](https://github.com/dvpombo/solete/blob/main/solete/modeling.py): `solete/modeling.py`
- [Forecast metrics](https://github.com/dvpombo/solete/blob/main/solete/metrics.py): `solete/metrics.py`
- [Forecast postprocessing](https://github.com/dvpombo/solete/blob/main/solete/postprocess.py): `solete/postprocess.py`

### Shared utilities

- [Data path resolution](https://github.com/dvpombo/solete/blob/main/solete/paths.py): `solete/paths.py`
- [Platform QC](https://github.com/dvpombo/solete/blob/main/solete/qc.py): `solete/qc.py`
- [PV physics](https://github.com/dvpombo/solete/blob/main/solete/physics.py): `solete/physics.py`

### Analysis and benchmarks

- [Dataset inspection](https://github.com/dvpombo/solete/blob/main/scripts/inspect_dataset.py): `scripts/inspect_dataset.py`
- [Forecast benchmarks](https://github.com/dvpombo/solete/tree/main/benchmarks): `benchmarks`

### Other components

- **Researcher**
- **Figshare dataset**
- [Raw and cleaned data](https://github.com/dvpombo/solete/tree/main/data/hdf5): `data/hdf5`

## Connections

- Researcher → Figshare dataset: downloads data (evidence: `README.md`)
- Figshare dataset → Raw and cleaned data: provides files (evidence: `README.md`)
- Researcher → 1-sec cleaning: runs cleaning (evidence: `README.md`)
- 1-sec cleaning → Cleaning QC rules: uses QC rules (evidence: `dataset/pipeline/clean_solete_1sec.py`)
- 1-sec cleaning → Solar position: computes solar position (evidence: `dataset/pipeline/clean_solete_1sec.py`)
- 1-sec cleaning → Pipeline reports: prints report (evidence: `dataset/pipeline/clean_solete_1sec.py`)
- 1-sec cleaning → Raw and cleaned data: reads and writes (evidence: `README.md`)
- Researcher → Resolution resampling: runs resampling (evidence: `README.md`)
- Resolution resampling → Raw and cleaned data: reads and writes (evidence: `README.md`)
- Resolution resampling → Pipeline reports: uses report helper (evidence: `dataset/pipeline/resample_solete.py`)
- Researcher → Parquet export: runs export (evidence: `README.md`)
- Parquet export → Raw and cleaned data: exports files (evidence: `README.md`)
- Researcher → Data diagnostics: investigates quality (evidence: `README.md`)
- Researcher → Forecast quickstart: configures forecasts (evidence: `README.md`)
- Forecast quickstart → Data loading and expansion: loads data (evidence: `scripts/quickstart/MLForecasting.py`)
- Forecast quickstart → Feature preparation: prepares features (evidence: `scripts/quickstart/MLForecasting.py`)
- Forecast quickstart → Forecast models: trains models (evidence: `scripts/quickstart/MLForecasting.py`)
- Forecast quickstart → Forecast postprocessing: processes predictions (evidence: `scripts/quickstart/MLForecasting.py`)
- Data loading and expansion → Data path resolution: resolves files (evidence: `solete/io.py`)
- Data loading and expansion → Feature preparation: expands data (evidence: `solete/io.py`)
- Data loading and expansion → Platform QC: applies QC (evidence: `solete/io.py`)
- Data loading and expansion → Forecast postprocessing: uses helpers (evidence: `solete/io.py`)
- Feature preparation → PV physics: uses physics features (evidence: `solete/preprocessing.py`)
- Feature preparation → Platform QC: uses QC precedence (evidence: `solete/preprocessing.py`)
- Forecast postprocessing → Data path resolution: writes outputs (evidence: `solete/postprocess.py`)
- Researcher → Dataset inspection: inspects dataset (evidence: `README.md`)
- Dataset inspection → Data path resolution: resolves data files (evidence: `scripts/inspect_dataset.py`)
- Researcher → Forecast benchmarks: reproduces benchmarks (evidence: `README.md`)
