How mtganiyu/arkham-data-engineer-challenge works
Arkham is a multi-tenant CPG and retail lakehouse that ingests ERP, partner POS, and public product/price data in ordered windows, then produces curated analytics, supply-network insights, and queryable metadata. The main workflow is window orchestration through Bronze ingestion, Silver normalization and quality handling, then Gold analytics; graph and catalog are additional product branches. README documents the layer-level flow. Source excerpts show key run behavior, but not every transformation or orchestration detail, so some documented subsystem links are omitted where direct source evidence is unavailable.
Main entry points: gold/run.py Gold analytics · bronze_run.py Bronze ingestion · orchestration.py Window orchestration · silver/run.py Silver processing
Components 26
Runtime and windows
- Window orchestration
src/arkham/orchestration.py - Runtime settings
src/arkham/config.py - Window registry
src/arkham/windows.py - Spark runtime
src/arkham/spark_session.py
Source ingestion
- ERP source
- Partner event API
legacy_sources/partner_api/main.py - Public data snapshots
- Bronze ingestion
src/arkham/ingest/bronze_run.py - Bronze Delta tables
src/arkham/ingest/bronze_writer.py
Lakehouse refinement
- Silver processing
src/arkham/silver/run.py - Silver transforms
src/arkham/silver - Silver Delta tables
src/arkham/silver/writer.py
Analytics and graph
- Gold analytics
src/arkham/gold/run.py - Sales and promotions
src/arkham/gold/commercial.py - Inventory analytics
src/arkham/gold/inventory.py - Gold Delta tables
src/arkham/gold/writer.py - Reconciliation and health
src/arkham/gold/reconciliation.py - Supply and product graph
src/arkham/graph/build.py - Graph exploration
src/arkham/graph/explore.py - Graph Delta tables
src/arkham/graph/build.py
Catalog and access
- Catalog refresh
src/arkham/catalog/build.py - Dataset declarations
src/arkham/catalog/registry.py - Catalog metadata
src/arkham/catalog/build.py - Catalog exploration
src/arkham/catalog/explore.py - SQL query access
src/arkham/query.py
Other components
- Pipeline operator
Connections 28
- Pipeline operator → Window orchestration: starts windows
- Window orchestration → Window registry: selects windows
src/arkham/orchestration.py - Window orchestration → Bronze ingestion: runs ingestion
src/arkham/orchestration.py - Window orchestration → Gold analytics: runs analytics
src/arkham/orchestration.py - Bronze ingestion → ERP source: reads source
- Bronze ingestion → Partner event API: fetches events
src/arkham/ingest/bronze_run.py - Bronze ingestion → Public data snapshots: loads snapshots
src/arkham/ingest/bronze_run.py - Bronze ingestion → Bronze Delta tables: writes windows
src/arkham/ingest/bronze_run.py - Silver processing → Bronze Delta tables: reads Bronze
src/arkham/silver/run.py - Silver processing → Silver transforms: applies transforms
src/arkham/silver/run.py - Silver processing → Silver Delta tables: writes curated data
src/arkham/silver/run.py - Gold analytics → Silver Delta tables: reads Silver
- Gold analytics → Sales and promotions: builds measures
src/arkham/gold/run.py - Gold analytics → Inventory analytics: builds positions
src/arkham/gold/run.py - Gold analytics → Gold Delta tables: writes Gold
src/arkham/gold/run.py - Gold analytics → Reconciliation and health: builds checks
src/arkham/gold/run.py - Supply and product graph → Silver Delta tables: reads facts
- Supply and product graph → Graph Delta tables: writes graph
src/arkham/graph/build.py - Graph exploration → Graph Delta tables: queries graph
src/arkham/graph/explore.py - Catalog refresh → Dataset declarations: combines declarations
src/arkham/catalog/build.py - Catalog refresh → Catalog metadata: writes metadata
src/arkham/catalog/build.py - Catalog exploration → Catalog metadata: explores metadata
- SQL query access → Dataset declarations: checks access
src/arkham/query.py - SQL query access → Gold Delta tables: queries tables
src/arkham/query.py - Runtime settings → Window orchestration: configures runtime
src/arkham/orchestration.py - Spark runtime → Bronze ingestion: provides Spark
src/arkham/ingest/bronze_run.py - Spark runtime → Silver processing: provides Spark
src/arkham/silver/run.py - Spark runtime → Gold analytics: provides Spark
src/arkham/gold/run.py