How avega17/ice-melt_ducklake works
Project Type & Purpose
• This is a research‐oriented modern data lakehouse pipeline (“ice-mELT DuckLake”) for Earth-Observation (EO) photovoltaic (PV) segmentation and energy-forecasting.
• It ingests vector (PV polygons/points) and raster (satellite imagery, irradiance) data, stores them in open formats, catalogs metadata in SQL, transforms via ELT (dbt & Hamilton), and serves through query engines and a documentation/dashboard site.
Key Architectural Layers & Components
A. External Data Sources
• DOI PV datasets (via datahugger & STAC)
• Public STAC catalogs for imagery/irradiance
• Overture Maps (admin boundaries, building footprints)
• Google Solar API / NREL NSRDB
B. Ingestion Layer
• Scripts & pipelines (doi_dataset_pipeline.py, raw_pv_doi_ingest.py)
• DLT or custom Python utilities (utils/ingestion_utils.py, utils/fetch_and_preprocess.py)
• Hamilton DAG orchestrators (db/validate_pv_ingest.py, test_hamilton_pipeline.py)
C. Raw Data Storage
• Local filesystem (or S3-compatible) buckets under datasets/raw/...
• Formats: GeoParquet (vector), COG/GeoTIFF (raster), Zarr/Icechunk (tensors)
D. Metadata & Catalog
• DuckLake metadata store (PostgreSQL or Neon/Supabase) via SQL tables
• Optional Apache Iceberg catalog for ACID table management
• STAC metadata catalog (pgstac, profiles.yml)
E. ELT/Transform Layer
• dbt project (dbt_project.yml + models/ folders)
• staging: raw standardization (models/staging)
• prepared: spatial processing, H3 indexing (models/prepared)
• curated: final analytical tables (models/curated)
• Python/SQL hybrid transformations (dbt Python models, Ibis, Xarray)
• Hamilton for programmatic DAGs
F. Analytical Engines
• DuckDB for local SQL and spatial extensions (dbt-duckdb)
• MotherDuck/DuckDB cloud scaling
• PostgreSQL for transactional metadata queries
G. Consumption & Visualization
• Documentation & dashboard (dashboard/ folder, Next.js or static site)
• Jupyter Notebooks (notebooks/) for EDA
• Future ML/forecasting models (in utils/torch_datasets.py, planned code)
Relationships & Data Flow
• Ingestion Layer writes Raw Data → Raw Data Storage
• Raw Data triggers metadata registration → Metadata & Catalog
• ELT Layer reads Raw Data + Metadata → writes Curated Tables → stored in DuckLake/Iceberg
• Analytical Engines query Curated Tables ←→ Metadata Store
• Outputs feed Dashboard & Notebooks
Architectural Patterns & Design Principles
• Modern ELT (dbt + Hamilton) over raw ingestion
• Data Lakehouse: separation of storage (object) vs metadata (SQL)
• Open formats: Parquet, GeoParquet, Zarr, Icechunk, COG
• Spatial indexing with H3 for performance
• Polyglot engine support via Ibis abstraction
• Free-tier/cloud-native cost optimization
Technologies & Frameworks (callouts)
• Ingestion: Python, DLT (if used), datahugger, Rustac/pgstac
• Storage: S3/R2, local FS, DuckLake (SQL), Apache Iceberg
• Catalog: PostgreSQL (Neon/Supabase), pgstac
• Transform: dbt (SQL + Python), Hamilton, Ibis, Xarray, Tensorstore
• Query Engines: DuckDB, MotherDuck, PostgreSQL
• Visualization: Next.js/MDX (dashboard/), Jupyter