# jiahong-que-9527/sololakehouse architecture

> Architecture diagram of the GitHub repository jiahong-que-9527/sololakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiahong-que-9527/sololakehouse
- Repository: https://github.com/jiahong-que-9527/sololakehouse

## Overview

SoloLakehouse is a local-first lakehouse reference platform for regulated data workflows. Its principal path collects ECB and EWG financial data, validates and stores Bronze records, transforms them through Silver into a Gold EUR-market dataset, and exposes governed data for SQL/BI and ML. Dagster coordinates assets and schedules; a governance layer supplies contracts, lineage, audit evidence, and operational controls. The README documents additional capabilities not fully visible in the sampled source; those are represented at their supported module paths, with uncertain wiring omitted.

Main entry points: [`dagster/assets.py`](https://github.com/jiahong-que-9527/sololakehouse/blob/main/dagster/assets.py) (Dagster assets), [`governance/lineage.py`](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/lineage.py) (Lineage evidence), [`ingestion/collectors/ecb_collector.py`](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_collector.py) (ECB collectors), [`ingestion/collectors/ecb_fx_collector.py`](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_fx_collector.py) (FX collector)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Data pipeline"]
  node_ecb_collector["ECB collectors<br/>[ecb_collector.py]"]
  node_fx_collector["FX collector"]
  node_ewg_collector["EWG collector<br/>[ewg_collector.py]"]
  node_bronze_checks["Bronze validation<br/>[bronze_checks.py]"]
  node_bronze_writer["Bronze writer<br/>[bronze_writer.py]"]
  node_silver_transforms["Silver transforms"]
  node_gold_transform["EUR market Gold"]
end

subgraph group_orchestration["Orchestration"]
  node_dagster_assets["Dagster assets<br/>[assets.py]"]
  node_dagster_defs["Dagster definitions<br/>[definitions.py]"]
end

subgraph group_consumption["Data products"]
  node_superset["Superset BI<br/>[superset_config.py]"]
  node_ml_training["ML experiments<br/>[evaluate.py]"]
end

subgraph group_governance["Governance"]
  node_contracts["Dataset contracts<br/>[contracts.py]"]
  node_lineage["Lineage evidence<br/>[lineage.py]"]
  node_audit["Audit evidence<br/>[audit.py]"]
  node_openmetadata_ingest["Metadata ingest"]
  node_interoperability["Catalog portability"]
  node_sovereignty["Sovereignty evidence<br/>[sovereignty.py]"]
  node_promotion["Promotion controls<br/>[promotion.py]"]
  node_operations["Operational evidence<br/>[operations.py]"]
  node_audit_storage["Audit storage<br/>[audit_storage.py]"]
end

subgraph group_platform["Platform services"]
  node_iceberg_io["Iceberg I/O<br/>[iceberg_io.py]"]
  node_iceberg_store[("Iceberg tables")]
  node_trino["Trino SQL"]
  node_ml_tracking[("MLflow")]
  node_openmetadata["OpenMetadata"]
  node_minio[("MinIO storage")]
end

node_operator(("Operator"))
node_ecb(("ECB data"))
node_ewg(("EWG data"))

node_operator -->|"launches runs"| node_dagster_defs
node_ecb -->|"supplies data"| node_ecb_collector
node_ecb -->|"supplies data"| node_fx_collector
node_ewg -->|"supplies data"| node_ewg_collector
node_dagster_assets -->|"invokes"| node_ecb_collector
node_dagster_assets -->|"invokes"| node_fx_collector
node_dagster_assets -->|"invokes"| node_ewg_collector
node_dagster_defs -->|"registers assets"| node_dagster_assets
node_ecb_collector -->|"validates records"| node_bronze_checks
node_fx_collector -->|"validates records"| node_bronze_checks
node_ewg_collector -->|"validates records"| node_bronze_checks
node_ecb_collector -->|"writes Bronze"| node_bronze_writer
node_fx_collector -->|"writes Bronze"| node_bronze_writer
node_ewg_collector -->|"writes Bronze"| node_bronze_writer
node_bronze_writer -->|"writes tables"| node_iceberg_io
node_iceberg_io -->|"reads and writes"| node_iceberg_store
node_iceberg_store -->|"stores objects"| node_minio
node_dagster_assets -->|"invokes"| node_silver_transforms
node_dagster_assets -->|"invokes"| node_gold_transform
node_silver_transforms -->|"reads and writes"| node_iceberg_io
node_gold_transform -->|"reads and writes"| node_iceberg_io
node_gold_transform -->|"uses Silver data"| node_silver_transforms
node_iceberg_store -->|"queried through"| node_trino
node_trino -->|"serves SQL"| node_superset
node_iceberg_store -->|"provides Gold snapshot"| node_ml_training
node_ml_training -->|"logs runs and artifacts"| node_ml_tracking
node_ml_training -->|"binds run evidence"| node_lineage
node_dagster_assets -->|"checks assets"| node_contracts
node_dagster_assets -->|"runs metadata sync"| node_openmetadata_ingest
node_openmetadata_ingest -->|"ingests metadata"| node_trino
node_openmetadata_ingest -->|"publishes metadata"| node_openmetadata
node_lineage -->|"joins catalog lineage"| node_openmetadata
node_lineage -->|"reads snapshots"| node_iceberg_store
node_lineage -->|"joins run evidence"| node_dagster_assets
node_lineage -->|"emits evidence"| node_audit
node_audit -->|"verifies retention"| node_audit_storage
node_audit_storage -->|"checks Object Lock"| node_minio
node_interoperability -.->|"proves catalog path"| node_iceberg_store
node_sovereignty -.->|"uses portability evidence"| node_interoperability
node_promotion -->|"records promotion evidence"| node_operations

click node_ecb_collector "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_collector.py"
click node_fx_collector "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_fx_collector.py"
click node_ewg_collector "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ewg_collector.py"
click node_bronze_checks "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/quality/bronze_checks.py"
click node_bronze_writer "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/bronze_writer.py"
click node_iceberg_io "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/iceberg_io.py"
click node_dagster_assets "https://github.com/jiahong-que-9527/sololakehouse/blob/main/dagster/assets.py"
click node_dagster_defs "https://github.com/jiahong-que-9527/sololakehouse/blob/main/dagster/definitions.py"
click node_silver_transforms "https://github.com/jiahong-que-9527/sololakehouse/tree/main/transformations"
click node_gold_transform "https://github.com/jiahong-que-9527/sololakehouse/blob/main/transformations/build_eur_market_daily.py"
click node_superset "https://github.com/jiahong-que-9527/sololakehouse/blob/main/docker/superset/superset_config.py"
click node_ml_training "https://github.com/jiahong-que-9527/sololakehouse/blob/main/ml/evaluate.py"
click node_contracts "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/contracts.py"
click node_lineage "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/lineage.py"
click node_audit "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/audit.py"
click node_openmetadata_ingest "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/openmetadata_ingest.py"
click node_interoperability "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/interoperability.py"
click node_sovereignty "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/sovereignty.py"
click node_promotion "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/promotion.py"
click node_operations "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/operations.py"
click node_audit_storage "https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/audit_storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ecb_collector,node_fx_collector,node_ewg_collector,node_bronze_checks,node_bronze_writer,node_silver_transforms,node_gold_transform toneBlue
class node_dagster_assets,node_dagster_defs toneAmber
class node_superset,node_ml_training toneMint
class node_contracts,node_lineage,node_audit,node_openmetadata_ingest,node_interoperability,node_sovereignty,node_promotion,node_operations,node_audit_storage toneRose
class node_iceberg_io,node_iceberg_store,node_trino,node_ml_tracking,node_openmetadata,node_minio,node_operator,node_ecb,node_ewg toneIndigo
```

## Components

### Data pipeline

- [ECB collectors](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_collector.py): `ingestion/collectors/ecb_collector.py`
- [FX collector](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ecb_fx_collector.py): `ingestion/collectors/ecb_fx_collector.py`
- [EWG collector](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/collectors/ewg_collector.py): `ingestion/collectors/ewg_collector.py`
- [Bronze validation](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/quality/bronze_checks.py): `ingestion/quality/bronze_checks.py`
- [Bronze writer](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/bronze_writer.py): `ingestion/bronze_writer.py`
- [Silver transforms](https://github.com/jiahong-que-9527/sololakehouse/tree/main/transformations): `transformations`
- [EUR market Gold](https://github.com/jiahong-que-9527/sololakehouse/blob/main/transformations/build_eur_market_daily.py): `transformations/build_eur_market_daily.py`

### Orchestration

- [Dagster assets](https://github.com/jiahong-que-9527/sololakehouse/blob/main/dagster/assets.py): `dagster/assets.py`
- [Dagster definitions](https://github.com/jiahong-que-9527/sololakehouse/blob/main/dagster/definitions.py): `dagster/definitions.py`

### Data products

- [Superset BI](https://github.com/jiahong-que-9527/sololakehouse/blob/main/docker/superset/superset_config.py): `docker/superset/superset_config.py`
- [ML experiments](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ml/evaluate.py): `ml/evaluate.py`

### Governance

- [Dataset contracts](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/contracts.py): `governance/contracts.py`
- [Lineage evidence](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/lineage.py): `governance/lineage.py`
- [Audit evidence](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/audit.py): `governance/audit.py`
- [Metadata ingest](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/openmetadata_ingest.py): `governance/openmetadata_ingest.py`
- [Catalog portability](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/interoperability.py): `governance/interoperability.py`
- [Sovereignty evidence](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/sovereignty.py): `governance/sovereignty.py`
- [Promotion controls](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/promotion.py): `governance/promotion.py`
- [Operational evidence](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/operations.py): `governance/operations.py`
- [Audit storage](https://github.com/jiahong-que-9527/sololakehouse/blob/main/governance/audit_storage.py): `governance/audit_storage.py`

### Platform services

- [Iceberg I/O](https://github.com/jiahong-que-9527/sololakehouse/blob/main/ingestion/iceberg_io.py): `ingestion/iceberg_io.py`
- **Iceberg tables**
- **Trino SQL**
- **MLflow**
- **OpenMetadata**
- **MinIO storage**

### Other components

- **Operator**
- **ECB data**
- **EWG data**

## Connections

- Operator → Dagster definitions: launches runs
- ECB data → ECB collectors: supplies data
- ECB data → FX collector: supplies data
- EWG data → EWG collector: supplies data
- Dagster assets → ECB collectors: invokes
- Dagster assets → FX collector: invokes
- Dagster assets → EWG collector: invokes
- Dagster definitions → Dagster assets: registers assets
- ECB collectors → Bronze validation: validates records
- FX collector → Bronze validation: validates records
- EWG collector → Bronze validation: validates records
- ECB collectors → Bronze writer: writes Bronze
- FX collector → Bronze writer: writes Bronze
- EWG collector → Bronze writer: writes Bronze
- Bronze writer → Iceberg I/O: writes tables
- Iceberg I/O → Iceberg tables: reads and writes
- Iceberg tables → MinIO storage: stores objects
- Dagster assets → Silver transforms: invokes
- Dagster assets → EUR market Gold: invokes
- Silver transforms → Iceberg I/O: reads and writes
- EUR market Gold → Iceberg I/O: reads and writes
- EUR market Gold → Silver transforms: uses Silver data
- Iceberg tables → Trino SQL: queried through
- Trino SQL → Superset BI: serves SQL
- Iceberg tables → ML experiments: provides Gold snapshot
- ML experiments → MLflow: logs runs and artifacts
- ML experiments → Lineage evidence: binds run evidence
- Dagster assets → Dataset contracts: checks assets
- Dagster assets → Metadata ingest: runs metadata sync
- Metadata ingest → Trino SQL: ingests metadata
- Metadata ingest → OpenMetadata: publishes metadata
- Lineage evidence → OpenMetadata: joins catalog lineage
- Lineage evidence → Iceberg tables: reads snapshots
- Lineage evidence → Dagster assets: joins run evidence
- Lineage evidence → Audit evidence: emits evidence
- Audit evidence → Audit storage: verifies retention
- Audit storage → MinIO storage: checks Object Lock
- Catalog portability → Iceberg tables: proves catalog path
- Sovereignty evidence → Catalog portability: uses portability evidence
- Promotion controls → Operational evidence: records promotion evidence