How javeed-pasha/learning-hub works
Learning Hub combines a static data-engineering learning site with a runnable Spark/Iceberg reference runtime. The documented flow applies framework and staging DDL, seeds source snapshots, runs metadata-driven processing into bronze/silver/gold, builds a sales-serving table, and can execute replay requests. The site also exposes topic pages and a runnable notebook. Runtime entry-point calls are shown in excerpts; other links follow explicit README workflow statements. Unsampled SQL, page, and notebook internals are not inferred.
Main entry points: runtime_framework.py Metadata runtime · run_pipeline.py Pipeline runner · replay_runner.py Replay runner · index.html Topic landing page
Components 16
Learning site
- Topic landing page
index.html - Topic articles
pages - Runtime guide
pages/working-reference-implementation.html - Demo notebook
notebooks/learning_hub_demo.ipynb
Demo data assets
- Framework DDL
ddl/demo_metadata_framework.sql - Staging DDL
ddl/demo_staging_sources.sql - Seed batches
ddl/demo_seed_batch_1001.sql - Second seed batch
ddl/demo_seed_batch_1002.sql - Sales serving SQL
ddl/demo_sales_serving.sql
Spark runtime
- Pipeline runner
spark/run_pipeline.py - Replay runner
spark/replay_runner.py - Metadata runtime
spark/runtime_framework.py
Runtime environment
- Spark-Iceberg stack
docker-compose.learning-hub.yml
Other components
- Learner
- Demo Iceberg catalog
- Jupyter service
Connections 16
- Learner → Topic landing page: opens
README.md - Topic landing page → Topic articles: links topics
README.md - Learner → Runtime guide: reads guide
README.md - Learner → Demo notebook: runs demo
README.md - Framework DDL → Demo Iceberg catalog: creates namespaces
README.md - Staging DDL → Demo Iceberg catalog: creates sources
README.md - Seed batches → Demo Iceberg catalog: seeds snapshot
README.md - Second seed batch → Demo Iceberg catalog: seeds changes
README.md - Pipeline runner → Metadata runtime: calls runtime
spark/run_pipeline.py - Replay runner → Metadata runtime: calls runtime
spark/replay_runner.py - Pipeline runner → Demo Iceberg catalog: processes sources
README.md - Metadata runtime → Demo Iceberg catalog: reads and writes
spark/runtime_framework.py - Sales serving SQL → Demo Iceberg catalog: builds sales output
README.md - Replay runner → Demo Iceberg catalog: executes rebuilds
README.md - Spark-Iceberg stack → Jupyter service: runs service
- Jupyter service → Demo notebook: hosts notebook
README.md