How jiahong-que-9527/sololakehouse works
SoloLakehouse is a local-first lakehouse reference platform for regulated data workflows. Its principal path collects ECB and EWG financial data, validates and stores Bronze records, transforms them through Silver into a Gold EUR-market dataset, and exposes governed data for SQL/BI and ML. Dagster coordinates assets and schedules; a governance layer supplies contracts, lineage, audit evidence, and operational controls. The README documents additional capabilities not fully visible in the sampled source; those are represented at their supported module paths, with uncertain wiring omitted.
Main entry points: assets.py Dagster assets · lineage.py Lineage evidence · ecb_collector.py ECB collectors · ecb_fx_collector.py FX collector
Components 29
Data pipeline
- ECB collectors
ingestion/collectors/ecb_collector.py - FX collector
ingestion/collectors/ecb_fx_collector.py - EWG collector
ingestion/collectors/ewg_collector.py - Bronze validation
ingestion/quality/bronze_checks.py - Bronze writer
ingestion/bronze_writer.py - Silver transforms
transformations - EUR market Gold
transformations/build_eur_market_daily.py
Orchestration
- Dagster assets
dagster/assets.py - Dagster definitions
dagster/definitions.py
Data products
- Superset BI
docker/superset/superset_config.py - ML experiments
ml/evaluate.py
Governance
- Dataset contracts
governance/contracts.py - Lineage evidence
governance/lineage.py - Audit evidence
governance/audit.py - Metadata ingest
governance/openmetadata_ingest.py - Catalog portability
governance/interoperability.py - Sovereignty evidence
governance/sovereignty.py - Promotion controls
governance/promotion.py - Operational evidence
governance/operations.py - Audit storage
governance/audit_storage.py
Platform services
- Iceberg I/O
ingestion/iceberg_io.py - Iceberg tables
- Trino SQL
- MLflow
- OpenMetadata
- MinIO storage
Other components
- Operator
- ECB data
- EWG data
Connections 40
- Operator → Dagster definitions: launches runs
- ECB data → ECB collectors: supplies data
- ECB data → FX collector: supplies data
- EWG data → EWG collector: supplies data
- Dagster assets → ECB collectors: invokes
- Dagster assets → FX collector: invokes
- Dagster assets → EWG collector: invokes
- Dagster definitions → Dagster assets: registers assets
- ECB collectors → Bronze validation: validates records
- FX collector → Bronze validation: validates records
- EWG collector → Bronze validation: validates records
- ECB collectors → Bronze writer: writes Bronze
- FX collector → Bronze writer: writes Bronze
- EWG collector → Bronze writer: writes Bronze
- Bronze writer → Iceberg I/O: writes tables
- Iceberg I/O → Iceberg tables: reads and writes
- Iceberg tables → MinIO storage: stores objects
- Dagster assets → Silver transforms: invokes
- Dagster assets → EUR market Gold: invokes
- Silver transforms → Iceberg I/O: reads and writes
- EUR market Gold → Iceberg I/O: reads and writes
- EUR market Gold → Silver transforms: uses Silver data
- Iceberg tables → Trino SQL: queried through
- Trino SQL → Superset BI: serves SQL
- Iceberg tables → ML experiments: provides Gold snapshot
- ML experiments → MLflow: logs runs and artifacts
- ML experiments → Lineage evidence: binds run evidence
- Dagster assets → Dataset contracts: checks assets
- Dagster assets → Metadata ingest: runs metadata sync
- Metadata ingest → Trino SQL: ingests metadata
- Metadata ingest → OpenMetadata: publishes metadata
- Lineage evidence → OpenMetadata: joins catalog lineage
- Lineage evidence → Iceberg tables: reads snapshots
- Lineage evidence → Dagster assets: joins run evidence
- Lineage evidence → Audit evidence: emits evidence
- Audit evidence → Audit storage: verifies retention
- Audit storage → MinIO storage: checks Object Lock
- Catalog portability → Iceberg tables: proves catalog path
- Sovereignty evidence → Catalog portability: uses portability evidence
- Promotion controls → Operational evidence: records promotion evidence