# mtganiyu/arkham-data-engineer-challenge architecture

> Architecture diagram of the GitHub repository mtganiyu/arkham-data-engineer-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mtganiyu/arkham-data-engineer-challenge
- Repository: https://github.com/mtganiyu/arkham-data-engineer-challenge

## Overview

Arkham is a multi-tenant CPG and retail lakehouse that ingests ERP, partner POS, and public product/price data in ordered windows, then produces curated analytics, supply-network insights, and queryable metadata. The main workflow is window orchestration through Bronze ingestion, Silver normalization and quality handling, then Gold analytics; graph and catalog are additional product branches. README documents the layer-level flow. Source excerpts show key run behavior, but not every transformation or orchestration detail, so some documented subsystem links are omitted where direct source evidence is unavailable.

Main entry points: [`src/arkham/gold/run.py`](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/run.py) (Gold analytics), [`src/arkham/ingest/bronze_run.py`](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/ingest/bronze_run.py) (Bronze ingestion), [`src/arkham/orchestration.py`](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/orchestration.py) (Window orchestration), [`src/arkham/silver/run.py`](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/silver/run.py) (Silver processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime and windows"]
  node_orchestration["Window orchestration<br/>[orchestration.py]"]
  node_settings["Runtime settings<br/>[config.py]"]
  node_windows["Window registry<br/>[windows.py]"]
  node_spark["Spark runtime<br/>[spark_session.py]"]
end

subgraph group_ingestion["Source ingestion"]
  node_erp[("ERP source")]
  node_partner["Partner event API<br/>[main.py]"]
  node_public["Public data snapshots"]
  node_bronze["Bronze ingestion<br/>[bronze_run.py]"]
  node_bronze_store[("Bronze Delta tables<br/>[bronze_writer.py]")]
end

subgraph group_refinement["Lakehouse refinement"]
  node_silver["Silver processing<br/>[run.py]"]
  node_silver_transforms["Silver transforms"]
  node_silver_store[("Silver Delta tables<br/>[writer.py]")]
end

subgraph group_insights["Analytics and graph"]
  node_gold["Gold analytics<br/>[run.py]"]
  node_gold_commercial["Sales and promotions<br/>[commercial.py]"]
  node_gold_inventory["Inventory analytics<br/>[inventory.py]"]
  node_gold_store[("Gold Delta tables<br/>[writer.py]")]
  node_reconciliation["Reconciliation and health<br/>[reconciliation.py]"]
  node_graph["Supply and product graph<br/>[build.py]"]
  node_graph_explore["Graph exploration<br/>[explore.py]"]
  node_graph_store[("Graph Delta tables<br/>[build.py]")]
end

subgraph group_discovery["Catalog and access"]
  node_catalog_build["Catalog refresh<br/>[build.py]"]
  node_catalog_registry["Dataset declarations<br/>[registry.py]"]
  node_catalog_store[("Catalog metadata<br/>[build.py]")]
  node_catalog_explore["Catalog exploration<br/>[explore.py]"]
  node_query["SQL query access<br/>[query.py]"]
end

node_operator(("Pipeline operator"))

node_operator -->|"starts windows"| node_orchestration
node_orchestration -->|"selects windows"| node_windows
node_orchestration -->|"runs ingestion"| node_bronze
node_orchestration -->|"runs analytics"| node_gold
node_bronze -->|"reads source"| node_erp
node_bronze -->|"fetches events"| node_partner
node_bronze -->|"loads snapshots"| node_public
node_bronze -->|"writes windows"| node_bronze_store
node_silver -->|"reads Bronze"| node_bronze_store
node_silver -->|"applies transforms"| node_silver_transforms
node_silver -->|"writes curated data"| node_silver_store
node_gold -->|"reads Silver"| node_silver_store
node_gold -->|"builds measures"| node_gold_commercial
node_gold -->|"builds positions"| node_gold_inventory
node_gold -->|"writes Gold"| node_gold_store
node_gold -->|"builds checks"| node_reconciliation
node_graph -->|"reads facts"| node_silver_store
node_graph -->|"writes graph"| node_graph_store
node_graph_explore -->|"queries graph"| node_graph_store
node_catalog_build -->|"combines declarations"| node_catalog_registry
node_catalog_build -->|"writes metadata"| node_catalog_store
node_catalog_explore -->|"explores metadata"| node_catalog_store
node_query -->|"checks access"| node_catalog_registry
node_query -->|"queries tables"| node_gold_store
node_settings -.->|"configures runtime"| node_orchestration
node_spark -.->|"provides Spark"| node_bronze
node_spark -.->|"provides Spark"| node_silver
node_spark -.->|"provides Spark"| node_gold

click node_orchestration "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/orchestration.py"
click node_settings "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/config.py"
click node_windows "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/windows.py"
click node_spark "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/spark_session.py"
click node_partner "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/legacy_sources/partner_api/main.py"
click node_bronze "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/ingest/bronze_run.py"
click node_bronze_store "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/ingest/bronze_writer.py"
click node_silver "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/silver/run.py"
click node_silver_transforms "https://github.com/mtganiyu/arkham-data-engineer-challenge/tree/main/src/arkham/silver"
click node_silver_store "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/silver/writer.py"
click node_gold "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/run.py"
click node_gold_commercial "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/commercial.py"
click node_gold_inventory "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/inventory.py"
click node_gold_store "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/writer.py"
click node_reconciliation "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/reconciliation.py"
click node_graph "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/build.py"
click node_graph_explore "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/explore.py"
click node_graph_store "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/build.py"
click node_catalog_build "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/build.py"
click node_catalog_registry "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/registry.py"
click node_catalog_store "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/build.py"
click node_catalog_explore "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/explore.py"
click node_query "https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/query.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestration,node_settings,node_windows,node_spark toneBlue
class node_erp,node_partner,node_public,node_bronze,node_bronze_store toneAmber
class node_silver,node_silver_transforms,node_silver_store toneMint
class node_gold,node_gold_commercial,node_gold_inventory,node_gold_store,node_reconciliation,node_graph,node_graph_explore,node_graph_store toneRose
class node_catalog_build,node_catalog_registry,node_catalog_store,node_catalog_explore,node_query,node_operator toneIndigo
```

## Components

### Runtime and windows

- [Window orchestration](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/orchestration.py): `src/arkham/orchestration.py`
- [Runtime settings](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/config.py): `src/arkham/config.py`
- [Window registry](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/windows.py): `src/arkham/windows.py`
- [Spark runtime](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/spark_session.py): `src/arkham/spark_session.py`

### Source ingestion

- **ERP source**
- [Partner event API](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/legacy_sources/partner_api/main.py): `legacy_sources/partner_api/main.py`
- **Public data snapshots**
- [Bronze ingestion](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/ingest/bronze_run.py): `src/arkham/ingest/bronze_run.py`
- [Bronze Delta tables](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/ingest/bronze_writer.py): `src/arkham/ingest/bronze_writer.py`

### Lakehouse refinement

- [Silver processing](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/silver/run.py): `src/arkham/silver/run.py`
- [Silver transforms](https://github.com/mtganiyu/arkham-data-engineer-challenge/tree/main/src/arkham/silver): `src/arkham/silver`
- [Silver Delta tables](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/silver/writer.py): `src/arkham/silver/writer.py`

### Analytics and graph

- [Gold analytics](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/run.py): `src/arkham/gold/run.py`
- [Sales and promotions](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/commercial.py): `src/arkham/gold/commercial.py`
- [Inventory analytics](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/inventory.py): `src/arkham/gold/inventory.py`
- [Gold Delta tables](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/writer.py): `src/arkham/gold/writer.py`
- [Reconciliation and health](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/gold/reconciliation.py): `src/arkham/gold/reconciliation.py`
- [Supply and product graph](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/build.py): `src/arkham/graph/build.py`
- [Graph exploration](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/explore.py): `src/arkham/graph/explore.py`
- [Graph Delta tables](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/graph/build.py): `src/arkham/graph/build.py`

### Catalog and access

- [Catalog refresh](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/build.py): `src/arkham/catalog/build.py`
- [Dataset declarations](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/registry.py): `src/arkham/catalog/registry.py`
- [Catalog metadata](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/build.py): `src/arkham/catalog/build.py`
- [Catalog exploration](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/catalog/explore.py): `src/arkham/catalog/explore.py`
- [SQL query access](https://github.com/mtganiyu/arkham-data-engineer-challenge/blob/main/src/arkham/query.py): `src/arkham/query.py`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Window orchestration: starts windows
- Window orchestration → Window registry: selects windows (evidence: `src/arkham/orchestration.py`)
- Window orchestration → Bronze ingestion: runs ingestion (evidence: `src/arkham/orchestration.py`)
- Window orchestration → Gold analytics: runs analytics (evidence: `src/arkham/orchestration.py`)
- Bronze ingestion → ERP source: reads source
- Bronze ingestion → Partner event API: fetches events (evidence: `src/arkham/ingest/bronze_run.py`)
- Bronze ingestion → Public data snapshots: loads snapshots (evidence: `src/arkham/ingest/bronze_run.py`)
- Bronze ingestion → Bronze Delta tables: writes windows (evidence: `src/arkham/ingest/bronze_run.py`)
- Silver processing → Bronze Delta tables: reads Bronze (evidence: `src/arkham/silver/run.py`)
- Silver processing → Silver transforms: applies transforms (evidence: `src/arkham/silver/run.py`)
- Silver processing → Silver Delta tables: writes curated data (evidence: `src/arkham/silver/run.py`)
- Gold analytics → Silver Delta tables: reads Silver
- Gold analytics → Sales and promotions: builds measures (evidence: `src/arkham/gold/run.py`)
- Gold analytics → Inventory analytics: builds positions (evidence: `src/arkham/gold/run.py`)
- Gold analytics → Gold Delta tables: writes Gold (evidence: `src/arkham/gold/run.py`)
- Gold analytics → Reconciliation and health: builds checks (evidence: `src/arkham/gold/run.py`)
- Supply and product graph → Silver Delta tables: reads facts
- Supply and product graph → Graph Delta tables: writes graph (evidence: `src/arkham/graph/build.py`)
- Graph exploration → Graph Delta tables: queries graph (evidence: `src/arkham/graph/explore.py`)
- Catalog refresh → Dataset declarations: combines declarations (evidence: `src/arkham/catalog/build.py`)
- Catalog refresh → Catalog metadata: writes metadata (evidence: `src/arkham/catalog/build.py`)
- Catalog exploration → Catalog metadata: explores metadata
- SQL query access → Dataset declarations: checks access (evidence: `src/arkham/query.py`)
- SQL query access → Gold Delta tables: queries tables (evidence: `src/arkham/query.py`)
- Runtime settings → Window orchestration: configures runtime (evidence: `src/arkham/orchestration.py`)
- Spark runtime → Bronze ingestion: provides Spark (evidence: `src/arkham/ingest/bronze_run.py`)
- Spark runtime → Silver processing: provides Spark (evidence: `src/arkham/silver/run.py`)
- Spark runtime → Gold analytics: provides Spark (evidence: `src/arkham/gold/run.py`)