# javeed-pasha/learning-hub architecture

> Architecture diagram of the GitHub repository javeed-pasha/learning-hub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/javeed-pasha/learning-hub
- Repository: https://github.com/javeed-pasha/learning-hub

## Overview

Learning Hub combines a static data-engineering learning site with a runnable Spark/Iceberg reference runtime. The documented flow applies framework and staging DDL, seeds source snapshots, runs metadata-driven processing into bronze/silver/gold, builds a sales-serving table, and can execute replay requests. The site also exposes topic pages and a runnable notebook. Runtime entry-point calls are shown in excerpts; other links follow explicit README workflow statements. Unsampled SQL, page, and notebook internals are not inferred.

Main entry points: [`spark/runtime_framework.py`](https://github.com/javeed-pasha/learning-hub/blob/main/spark/runtime_framework.py) (Metadata runtime), [`spark/run_pipeline.py`](https://github.com/javeed-pasha/learning-hub/blob/main/spark/run_pipeline.py) (Pipeline runner), [`spark/replay_runner.py`](https://github.com/javeed-pasha/learning-hub/blob/main/spark/replay_runner.py) (Replay runner), [`index.html`](https://github.com/javeed-pasha/learning-hub/blob/main/index.html) (Topic landing page)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_site["Learning site"]
  node_landing["Topic landing page<br/>[index.html]"]
  node_articles["Topic articles"]
  node_reference["Runtime guide"]
  node_notebook["Demo notebook"]
end

subgraph group_assets["Demo data assets"]
  node_framework_ddl["Framework DDL"]
  node_staging_ddl["Staging DDL"]
  node_seed_ddl["Seed batches"]
  node_seed2_ddl["Second seed batch"]
  node_serving_ddl["Sales serving SQL"]
end

subgraph group_runtime["Spark runtime"]
  node_pipeline["Pipeline runner<br/>[run_pipeline.py]"]
  node_replay["Replay runner<br/>[replay_runner.py]"]
  node_framework["Metadata runtime"]
end

subgraph group_environment["Runtime environment"]
  node_compose["Spark-Iceberg stack"]
end

node_learner(("Learner"))
node_catalog[("Demo Iceberg catalog")]
node_jupyter["Jupyter service"]

node_learner -->|"opens"| node_landing
node_landing -->|"links topics"| node_articles
node_learner -->|"reads guide"| node_reference
node_learner -->|"runs demo"| node_notebook
node_framework_ddl -->|"creates namespaces"| node_catalog
node_staging_ddl -->|"creates sources"| node_catalog
node_seed_ddl -->|"seeds snapshot"| node_catalog
node_seed2_ddl -->|"seeds changes"| node_catalog
node_pipeline -->|"calls runtime"| node_framework
node_replay -->|"calls runtime"| node_framework
node_pipeline -->|"processes sources"| node_catalog
node_framework -->|"reads and writes"| node_catalog
node_serving_ddl -->|"builds sales output"| node_catalog
node_replay -->|"executes rebuilds"| node_catalog
node_compose -.->|"runs service"| node_jupyter
node_jupyter -->|"hosts notebook"| node_notebook

click node_landing "https://github.com/javeed-pasha/learning-hub/blob/main/index.html"
click node_articles "https://github.com/javeed-pasha/learning-hub/tree/main/pages"
click node_reference "https://github.com/javeed-pasha/learning-hub/blob/main/pages/working-reference-implementation.html"
click node_notebook "https://github.com/javeed-pasha/learning-hub/blob/main/notebooks/learning_hub_demo.ipynb"
click node_framework_ddl "https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_metadata_framework.sql"
click node_staging_ddl "https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_staging_sources.sql"
click node_seed_ddl "https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_seed_batch_1001.sql"
click node_seed2_ddl "https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_seed_batch_1002.sql"
click node_serving_ddl "https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_sales_serving.sql"
click node_pipeline "https://github.com/javeed-pasha/learning-hub/blob/main/spark/run_pipeline.py"
click node_replay "https://github.com/javeed-pasha/learning-hub/blob/main/spark/replay_runner.py"
click node_framework "https://github.com/javeed-pasha/learning-hub/blob/main/spark/runtime_framework.py"
click node_compose "https://github.com/javeed-pasha/learning-hub/blob/main/docker-compose.learning-hub.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_landing,node_articles,node_reference,node_notebook toneBlue
class node_framework_ddl,node_staging_ddl,node_seed_ddl,node_seed2_ddl,node_serving_ddl,node_catalog toneAmber
class node_pipeline,node_replay,node_framework toneMint
class node_compose toneRose
class node_learner,node_jupyter toneIndigo
```

## Components

### Learning site

- [Topic landing page](https://github.com/javeed-pasha/learning-hub/blob/main/index.html): `index.html`
- [Topic articles](https://github.com/javeed-pasha/learning-hub/tree/main/pages): `pages`
- [Runtime guide](https://github.com/javeed-pasha/learning-hub/blob/main/pages/working-reference-implementation.html): `pages/working-reference-implementation.html`
- [Demo notebook](https://github.com/javeed-pasha/learning-hub/blob/main/notebooks/learning_hub_demo.ipynb): `notebooks/learning_hub_demo.ipynb`

### Demo data assets

- [Framework DDL](https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_metadata_framework.sql): `ddl/demo_metadata_framework.sql`
- [Staging DDL](https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_staging_sources.sql): `ddl/demo_staging_sources.sql`
- [Seed batches](https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_seed_batch_1001.sql): `ddl/demo_seed_batch_1001.sql`
- [Second seed batch](https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_seed_batch_1002.sql): `ddl/demo_seed_batch_1002.sql`
- [Sales serving SQL](https://github.com/javeed-pasha/learning-hub/blob/main/ddl/demo_sales_serving.sql): `ddl/demo_sales_serving.sql`

### Spark runtime

- [Pipeline runner](https://github.com/javeed-pasha/learning-hub/blob/main/spark/run_pipeline.py): `spark/run_pipeline.py`
- [Replay runner](https://github.com/javeed-pasha/learning-hub/blob/main/spark/replay_runner.py): `spark/replay_runner.py`
- [Metadata runtime](https://github.com/javeed-pasha/learning-hub/blob/main/spark/runtime_framework.py): `spark/runtime_framework.py`

### Runtime environment

- [Spark-Iceberg stack](https://github.com/javeed-pasha/learning-hub/blob/main/docker-compose.learning-hub.yml): `docker-compose.learning-hub.yml`

### Other components

- **Learner**
- **Demo Iceberg catalog**
- **Jupyter service**

## Connections

- Learner → Topic landing page: opens (evidence: `README.md`)
- Topic landing page → Topic articles: links topics (evidence: `README.md`)
- Learner → Runtime guide: reads guide (evidence: `README.md`)
- Learner → Demo notebook: runs demo (evidence: `README.md`)
- Framework DDL → Demo Iceberg catalog: creates namespaces (evidence: `README.md`)
- Staging DDL → Demo Iceberg catalog: creates sources (evidence: `README.md`)
- Seed batches → Demo Iceberg catalog: seeds snapshot (evidence: `README.md`)
- Second seed batch → Demo Iceberg catalog: seeds changes (evidence: `README.md`)
- Pipeline runner → Metadata runtime: calls runtime (evidence: `spark/run_pipeline.py`)
- Replay runner → Metadata runtime: calls runtime (evidence: `spark/replay_runner.py`)
- Pipeline runner → Demo Iceberg catalog: processes sources (evidence: `README.md`)
- Metadata runtime → Demo Iceberg catalog: reads and writes (evidence: `spark/runtime_framework.py`)
- Sales serving SQL → Demo Iceberg catalog: builds sales output (evidence: `README.md`)
- Replay runner → Demo Iceberg catalog: executes rebuilds (evidence: `README.md`)
- Spark-Iceberg stack → Jupyter service: runs service
- Jupyter service → Demo notebook: hosts notebook (evidence: `README.md`)