# kadrisof/grid-pulse architecture

> Architecture diagram of the GitHub repository kadrisof/grid-pulse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kadrisof/grid-pulse
- Repository: https://github.com/kadrisof/grid-pulse

## Overview

GridPulse ingests ENTSO-E electricity-market data through scheduled Airflow tasks. The main workflow fetches raw data, stores Bronze objects in MinIO, and emits asset metadata; a triggered consumer fetches structured data and writes Silver Parquet. A parallel Bronze task standardizes structured data and upserts supported datasets to PostgreSQL. The repository also includes dbt staging models for seven market datasets. README content was empty, so capabilities and relationships here are limited to the tree and shown source references.

Main entry points: [`dags/entsoe_bronze_producer.py`](https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_bronze_producer.py) (Bronze producer), [`dags/entsoe_silver_consumer.py`](https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_silver_consumer.py) (Silver consumer), [`src/gridpulse/ingestion/entsoe_service.py`](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/entsoe_service.py) (ENTSO-E service), [`src/gridpulse/storage/minio_client.py`](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/minio_client.py) (MinIO client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Airflow orchestration"]
  node_bronze_dag["Bronze producer"]
  node_silver_dag["Silver consumer"]
end

subgraph group_ingest["ENTSO-E ingestion"]
  node_dataset_config["Dataset configuration<br/>[config.py]"]
  node_entsoe_service["ENTSO-E service<br/>[entsoe_service.py]"]
  node_entsoe_client["ENTSO-E client<br/>[entsoe_client.py]"]
  node_entsoe_mappings["ENTSO-E mappings<br/>[entsoe_mapping.py]"]
end

subgraph group_storage["Data storage"]
  node_persistence["Postgres persistence"]
  node_db_models["Relational models<br/>[db_models.py]"]
  node_postgres_client["Postgres client<br/>[postgres_client.py]"]
  node_minio_client["MinIO client<br/>[minio_client.py]"]
  node_bronze_objects["Bronze objects<br/>[minio_client.py]"]
  node_silver_objects["Silver Parquet"]
end

subgraph group_transform["Data transformation"]
  node_standardizer["Data standardizer"]
  node_dbt_staging["Market staging models"]
end

node_scheduler(("Airflow scheduler"))
node_entsoe_api{{"ENTSO-E API"}}
node_postgres[("PostgreSQL")]
node_minio[("MinIO object store")]

node_scheduler -->|"schedules"| node_bronze_dag
node_bronze_dag -->|"reads datasets"| node_dataset_config
node_bronze_dag -->|"fetches data"| node_entsoe_service
node_entsoe_service -->|"reads settings"| node_dataset_config
node_entsoe_service -->|"calls SDK"| node_entsoe_client
node_entsoe_client -->|"requests data"| node_entsoe_api
node_bronze_dag -->|"stores raw data"| node_minio_client
node_minio_client -->|"uploads objects"| node_minio
node_minio_client -->|"writes Bronze"| node_bronze_objects
node_bronze_dag -->|"standardizes data"| node_standardizer
node_bronze_dag -->|"upserts rows"| node_persistence
node_persistence -->|"selects model"| node_db_models
node_persistence -->|"writes rows"| node_postgres_client
node_postgres_client -->|"connects to"| node_postgres
node_bronze_dag -->|"emits asset events"| node_silver_dag
node_silver_dag -->|"fetches structured data"| node_entsoe_service
node_silver_dag -->|"stores Parquet"| node_minio_client
node_silver_dag -->|"produces Silver"| node_silver_objects
node_silver_dag -->|"writes objects"| node_minio

click node_bronze_dag "https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_bronze_producer.py"
click node_silver_dag "https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_silver_consumer.py"
click node_dataset_config "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/config.py"
click node_entsoe_service "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/entsoe_service.py"
click node_entsoe_client "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/api/entsoe_client.py"
click node_standardizer "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/data_standardizer.py"
click node_persistence "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/persistence_service.py"
click node_db_models "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/models/db_models.py"
click node_postgres_client "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/postgres_client.py"
click node_minio_client "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/minio_client.py"
click node_bronze_objects "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/minio_client.py"
click node_silver_objects "https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_silver_consumer.py"
click node_entsoe_mappings "https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/utils/entsoe_mapping.py"
click node_dbt_staging "https://github.com/kadrisof/grid-pulse/tree/main/dbt/models/staging"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bronze_dag,node_silver_dag toneBlue
class node_dataset_config,node_entsoe_service,node_entsoe_client,node_entsoe_mappings,node_postgres,node_minio toneAmber
class node_persistence,node_db_models,node_postgres_client,node_minio_client,node_bronze_objects,node_silver_objects,node_entsoe_api toneMint
class node_standardizer,node_dbt_staging,node_scheduler toneRose
```

## Components

### Airflow orchestration

- [Bronze producer](https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_bronze_producer.py): `dags/entsoe_bronze_producer.py`
- [Silver consumer](https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_silver_consumer.py): `dags/entsoe_silver_consumer.py`

### ENTSO-E ingestion

- [Dataset configuration](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/config.py): `src/gridpulse/config.py`
- [ENTSO-E service](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/entsoe_service.py): `src/gridpulse/ingestion/entsoe_service.py`
- [ENTSO-E client](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/api/entsoe_client.py): `src/gridpulse/api/entsoe_client.py`
- [ENTSO-E mappings](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/utils/entsoe_mapping.py): `src/gridpulse/utils/entsoe_mapping.py`

### Data storage

- [Postgres persistence](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/persistence_service.py): `src/gridpulse/ingestion/persistence_service.py`
- [Relational models](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/models/db_models.py): `src/gridpulse/models/db_models.py`
- [Postgres client](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/postgres_client.py): `src/gridpulse/storage/postgres_client.py`
- [MinIO client](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/minio_client.py): `src/gridpulse/storage/minio_client.py`
- [Bronze objects](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/storage/minio_client.py): `src/gridpulse/storage/minio_client.py`
- [Silver Parquet](https://github.com/kadrisof/grid-pulse/blob/main/dags/entsoe_silver_consumer.py): `dags/entsoe_silver_consumer.py`

### Data transformation

- [Data standardizer](https://github.com/kadrisof/grid-pulse/blob/main/src/gridpulse/ingestion/data_standardizer.py): `src/gridpulse/ingestion/data_standardizer.py`
- [Market staging models](https://github.com/kadrisof/grid-pulse/tree/main/dbt/models/staging): `dbt/models/staging`

### Other components

- **Airflow scheduler**
- **ENTSO-E API**
- **PostgreSQL**
- **MinIO object store**

## Connections

- Airflow scheduler → Bronze producer: schedules (evidence: `dags/entsoe_bronze_producer.py`)
- Bronze producer → Dataset configuration: reads datasets (evidence: `dags/entsoe_bronze_producer.py`)
- Bronze producer → ENTSO-E service: fetches data (evidence: `dags/entsoe_bronze_producer.py`)
- ENTSO-E service → Dataset configuration: reads settings (evidence: `src/gridpulse/ingestion/entsoe_service.py`)
- ENTSO-E service → ENTSO-E client: calls SDK (evidence: `src/gridpulse/ingestion/entsoe_service.py`)
- ENTSO-E client → ENTSO-E API: requests data (evidence: `src/gridpulse/api/entsoe_client.py`)
- Bronze producer → MinIO client: stores raw data (evidence: `dags/entsoe_bronze_producer.py`)
- MinIO client → MinIO object store: uploads objects (evidence: `src/gridpulse/storage/minio_client.py`)
- MinIO client → Bronze objects: writes Bronze (evidence: `src/gridpulse/storage/minio_client.py`)
- Bronze producer → Data standardizer: standardizes data (evidence: `dags/entsoe_bronze_producer.py`)
- Bronze producer → Postgres persistence: upserts rows (evidence: `dags/entsoe_bronze_producer.py`)
- Postgres persistence → Relational models: selects model (evidence: `src/gridpulse/ingestion/persistence_service.py`)
- Postgres persistence → Postgres client: writes rows (evidence: `src/gridpulse/ingestion/persistence_service.py`)
- Postgres client → PostgreSQL: connects to (evidence: `src/gridpulse/storage/postgres_client.py`)
- Bronze producer → Silver consumer: emits asset events (evidence: `dags/entsoe_bronze_producer.py`)
- Silver consumer → ENTSO-E service: fetches structured data (evidence: `dags/entsoe_silver_consumer.py`)
- Silver consumer → MinIO client: stores Parquet (evidence: `dags/entsoe_silver_consumer.py`)
- Silver consumer → Silver Parquet: produces Silver (evidence: `dags/entsoe_silver_consumer.py`)
- Silver consumer → MinIO object store: writes objects (evidence: `dags/entsoe_silver_consumer.py`)
