How kadrisof/grid-pulse works
GridPulse ingests ENTSO-E electricity-market data through scheduled Airflow tasks. The main workflow fetches raw data, stores Bronze objects in MinIO, and emits asset metadata; a triggered consumer fetches structured data and writes Silver Parquet. A parallel Bronze task standardizes structured data and upserts supported datasets to PostgreSQL. The repository also includes dbt staging models for seven market datasets. README content was empty, so capabilities and relationships here are limited to the tree and shown source references.
Main entry points: entsoe_bronze_producer.py Bronze producer · entsoe_silver_consumer.py Silver consumer · entsoe_service.py ENTSO-E service · minio_client.py MinIO client
Components 18
Airflow orchestration
- Bronze producer
dags/entsoe_bronze_producer.py - Silver consumer
dags/entsoe_silver_consumer.py
ENTSO-E ingestion
- Dataset configuration
src/gridpulse/config.py - ENTSO-E service
src/gridpulse/ingestion/entsoe_service.py - ENTSO-E client
src/gridpulse/api/entsoe_client.py - ENTSO-E mappings
src/gridpulse/utils/entsoe_mapping.py
Data storage
- Postgres persistence
src/gridpulse/ingestion/persistence_service.py - Relational models
src/gridpulse/models/db_models.py - Postgres client
src/gridpulse/storage/postgres_client.py - MinIO client
src/gridpulse/storage/minio_client.py - Bronze objects
src/gridpulse/storage/minio_client.py - Silver Parquet
dags/entsoe_silver_consumer.py
Data transformation
- Data standardizer
src/gridpulse/ingestion/data_standardizer.py - Market staging models
dbt/models/staging
Other components
- Airflow scheduler
- ENTSO-E API
- PostgreSQL
- MinIO object store
Connections 19
- Airflow scheduler → Bronze producer: schedules
dags/entsoe_bronze_producer.py - Bronze producer → Dataset configuration: reads datasets
dags/entsoe_bronze_producer.py - Bronze producer → ENTSO-E service: fetches data
dags/entsoe_bronze_producer.py - ENTSO-E service → Dataset configuration: reads settings
src/gridpulse/ingestion/entsoe_service.py - ENTSO-E service → ENTSO-E client: calls SDK
src/gridpulse/ingestion/entsoe_service.py - ENTSO-E client → ENTSO-E API: requests data
src/gridpulse/api/entsoe_client.py - Bronze producer → MinIO client: stores raw data
dags/entsoe_bronze_producer.py - MinIO client → MinIO object store: uploads objects
src/gridpulse/storage/minio_client.py - MinIO client → Bronze objects: writes Bronze
src/gridpulse/storage/minio_client.py - Bronze producer → Data standardizer: standardizes data
dags/entsoe_bronze_producer.py - Bronze producer → Postgres persistence: upserts rows
dags/entsoe_bronze_producer.py - Postgres persistence → Relational models: selects model
src/gridpulse/ingestion/persistence_service.py - Postgres persistence → Postgres client: writes rows
src/gridpulse/ingestion/persistence_service.py - Postgres client → PostgreSQL: connects to
src/gridpulse/storage/postgres_client.py - Bronze producer → Silver consumer: emits asset events
dags/entsoe_bronze_producer.py - Silver consumer → ENTSO-E service: fetches structured data
dags/entsoe_silver_consumer.py - Silver consumer → MinIO client: stores Parquet
dags/entsoe_silver_consumer.py - Silver consumer → Silver Parquet: produces Silver
dags/entsoe_silver_consumer.py - Silver consumer → MinIO object store: writes objects
dags/entsoe_silver_consumer.py