# satyakiguha007/retail-data-platform architecture

> Architecture diagram of the GitHub repository satyakiguha007/retail-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/satyakiguha007/retail-data-platform
- Repository: https://github.com/satyakiguha007/retail-data-platform

## Overview

1. **Overall shape**
   - This repo is a retail data platform on Azure + Databricks, organized as a staged pipeline: source simulation/collection → bronze ingestion → silver conformance and sales audit → gold marts/metrics → optional LLM analytics.
   - The main architectural boundary is between **data-producing code in Python** and **lakehouse transformation code in Databricks-style scripts/notebooks**.

2. **Source systems and generators**
   - `pos_simulator/` is the primary synthetic operational source: it produces point-of-sale transaction logs as partitioned `ndjson` files by `store/date/hour`.
   - `ingestion/` contains additional source-side generators/fetchers for **FX rates**, **weather**, **marketplace feeds**, and **Olist** reference/e-commerce data.
   - These sources are independent of the lakehouse logic; they create raw artifacts that later feed ingestion jobs.

3. **Raw data layout**
   - The repository includes representative landed data under `output/` and `data/landing/`.
   - `output/mkt_feed/.../feed.ndjson` and `output/pos_rtlog/.../rtlog.ndjson` show the expected raw partitioning pattern.
   - `data/landing/olist/` holds CSV inputs for the Olist dataset, indicating the platform supports both file-based batch inputs and generated event-style feeds.

4. **Operator/control plane**
   - `adls_sync_console/` is a separate Streamlit-based console used to sync data to ADLS and manage uploads/scheduling/statistics.
   - Its core modules (`adls.py`, `config.py`, `scheduler.py`, `state.py`, `sync.py`, `validate.py`) suggest it acts as the human-facing operational boundary for moving local files into cloud storage and tracking sync history (`data/history.json`).

5. **Bronze ingestion layer**
   - `transformations/bronze/` contains the first Databricks processing stage, with one job per source: FX rates, weather, POS RT logs, marketplace, Olist, stores, and SA store data.
   - This layer is responsible for getting raw source files into a standardized bronze representation, likely with source-specific validation and minimal reshaping.
   - The presence of notebooks and Python scripts suggests mixed interactive and job-oriented execution.

6. **Cross-cutting ingestion safeguards**
   - `transformations/silver/_shared/` includes reusable controls such as `schema_gate.py`, `quarantine.py`, `fx_helpers.py`, and `surrogate_keys.py`.
   - These indicate the platform enforces schema/quality gates and separates rejected records from clean records before conformance.
   - This is an important boundary for the diagram: **bronze can accept messy source data; silver only accepts validated/conformed data**.

7. **Silver conformance model**
   - `transformations/silver/` is centered on an Oracle ReSA-style canonical sales model.
   - The folder structure mirrors the business entities: transaction header, item, discount, tender, tax, inter-company tax, store-day, and seller data.
   - Separate source adapters exist for marketplace, POS, Olist, and seller/store-related feeds, showing that multiple channels are normalized into one sales-audit schema.

8. **Sales audit subsystem**
   - `transformations/silver/audit/` is a distinct reconciliation layer with a rule framework and 18 specific rules under `rules/`.
   - The rules cover count, total, foreign-key, sign, validation, outlier, and business-date consistency checks, plus currency alignment.
   - `sa_error_writer.py` and the shared rule framework imply audit failures are persisted as first-class data, not just logs.

9. **Gold analytics layer**
   - `transformations/gold/` builds downstream dimensions, facts, and marts from the conformed/audited silver data.
   - Key outputs include `dim_date`, `dim_channel`, `dim_store`, `dim_tender`, `dim_item`, and facts such as `fact_sales_line`, `fact_tender`, `fact_store_day`, plus marts for audit summary, channel P&L, store-day scorecards, and payment mix.
   - Shared helpers like `cdf_reader.py`, `dim_lookup.py`, and `watermark.py` suggest incremental processing and dimension lookup patterns.

10. **LLM layer**
   - `llm/` is a separate top-layer capability with placeholders for review enrichment, text-to-SQL, weekly narrative, and client integrations.
   - Its location shows it consumes curated warehouse outputs rather than raw feeds, so it belongs above gold in the architecture.
   - The repo tree suggests this layer is scaffolded but not yet populated with implementation.

11. **Infrastructure and delivery**
   - `infra/` contains Terraform environment scaffolding for `dev` and `prod`, so cloud resources are managed as code.
   - The README also calls out GitHub Actions CI/CD and Databricks Asset Bundles, implying deployment automation into Databricks and Azure storage/runtime resources.
   - `scripts/test_adls_connection.py` provides a small operational check for Azure storage connectivity.

12. **Testing and sample fixtures**
   - `tests/pos_simulator/` validates the simulator’s generator, faults, and writer behavior.
   - `sample_data/` directories under the simulator and ingestion sources provide reproducible fixtures for local development and smoke tests.
   - This supports a workflow where local generation can be verified before data is promoted into the lakehouse pipeline.

13. **Diagram boundaries to preserve**
   - Treat the repo as four layers: **source generation/control plane**, **raw landing/bronze**, **conformed silver with audit**, and **serving gold + LLM analytics**.
   - The critical data-flow arrows are:
     - source generators → landed files/ADLS
     - bronze jobs → validated/conformed silver tables
     - silver audit rules → audit error outputs
     - silver/gold outputs → analytics marts and LLM consumers
   - The main operational boundary is the Streamlit sync console; the main analytical boundary is the Databricks lakehouse.

Main entry points: [`adls_sync_console`](https://github.com/satyakiguha007/retail-data-platform/tree/main/adls_sync_console) (ADLS console), [`transformations/silver/_shared`](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver/_shared) (Silver controls), [`transformations/silver`](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver) (Sales conformance), [`infra`](https://github.com/satyakiguha007/retail-data-platform/tree/main/infra) (Infra)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_sources["Sources"]
  node_n_pos_sim["POS simulator<br/>python source generator"]
  node_n_ingestion["Batch sources<br/>source fetchers"]
  node_n_landing["Landing data<br/>raw fixtures"]
end

subgraph group_g_ops["Operations"]
  node_n_sync["ADLS console<br/>streamlit ops console"]
end

subgraph group_g_lakehouse["Lakehouse"]
  node_n_bronze_fx["FX bronze<br/>bronze job<br/>[01_fx_rates.py]"]
  node_n_bronze_weather["Weather bronze<br/>bronze job"]
  node_n_bronze_pos["POS bronze<br/>bronze job<br/>[03_pos_rtlog.py]"]
  node_n_bronze_mkt["Marketplace bronze<br/>bronze job<br/>[04_marketplace.py]"]
  node_n_bronze_olist["Olist bronze<br/>bronze job<br/>[05_olist.py]"]
  node_n_silver_shared["Silver controls<br/>shared conformance"]
  node_n_silver_sales["Sales conformance<br/>canonical sales model"]
  node_n_silver_audit["Sales audit<br/>reconciliation layer"]
  node_n_gold_dims["Dimensions<br/>gold marts"]
  node_n_gold_facts["Facts<br/>gold marts"]
  node_n_gold_marts["Gold marts<br/>analytics marts"]
end

subgraph group_g_serving["Serving"]
  node_n_llm["LLM layer<br/>consumer analytics"]
end

subgraph group_g_platform["Platform"]
  node_n_infra["Infra<br/>terraform environments"]
  node_n_tests["Simulator tests<br/>pytest suite"]
end

node_n_pos_sim -->|"local files"| node_n_sync
node_n_ingestion -->|"local files"| node_n_sync
node_n_landing -->|"batch input"| node_n_bronze_olist
node_n_sync -->|"lands to ADLS"| node_n_bronze_fx
node_n_sync -->|"lands to ADLS"| node_n_bronze_weather
node_n_sync -->|"lands to ADLS"| node_n_bronze_pos
node_n_sync -->|"lands to ADLS"| node_n_bronze_mkt
node_n_sync -->|"lands to ADLS"| node_n_bronze_olist
node_n_bronze_fx -->|"validate"| node_n_silver_shared
node_n_bronze_weather -->|"validate"| node_n_silver_shared
node_n_bronze_pos -->|"validate"| node_n_silver_shared
node_n_bronze_mkt -->|"validate"| node_n_silver_shared
node_n_bronze_olist -->|"validate"| node_n_silver_shared
node_n_silver_shared -->|"conform"| node_n_silver_sales
node_n_silver_sales -->|"audit"| node_n_silver_audit
node_n_silver_audit -->|"error fact"| node_n_gold_facts
node_n_silver_sales -->|"dimensions"| node_n_gold_dims
node_n_silver_sales -->|"facts"| node_n_gold_facts
node_n_gold_dims -->|"lookup"| node_n_gold_marts
node_n_gold_facts -->|"aggregate"| node_n_gold_marts
node_n_gold_marts -->|"curated context"| node_n_llm
node_n_infra -.->|"deploy support"| node_n_sync
node_n_infra -.->|"deploy"| node_n_bronze_fx
node_n_infra -.->|"deploy"| node_n_silver_sales
node_n_infra -.->|"deploy"| node_n_gold_marts
node_n_tests -.->|"verify"| node_n_pos_sim

click node_n_pos_sim "https://github.com/satyakiguha007/retail-data-platform/tree/main/pos_simulator"
click node_n_ingestion "https://github.com/satyakiguha007/retail-data-platform/tree/main/ingestion"
click node_n_landing "https://github.com/satyakiguha007/retail-data-platform/tree/main/data/landing/olist"
click node_n_sync "https://github.com/satyakiguha007/retail-data-platform/tree/main/adls_sync_console"
click node_n_bronze_fx "https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/01_fx_rates.py"
click node_n_bronze_weather "https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/02_weather.py.ipynb"
click node_n_bronze_pos "https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/03_pos_rtlog.py"
click node_n_bronze_mkt "https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/04_marketplace.py"
click node_n_bronze_olist "https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/05_olist.py"
click node_n_silver_shared "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver/_shared"
click node_n_silver_sales "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver"
click node_n_silver_audit "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver/audit"
click node_n_gold_dims "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold"
click node_n_gold_facts "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold"
click node_n_gold_marts "https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold"
click node_n_llm "https://github.com/satyakiguha007/retail-data-platform/tree/main/llm"
click node_n_infra "https://github.com/satyakiguha007/retail-data-platform/tree/main/infra"
click node_n_tests "https://github.com/satyakiguha007/retail-data-platform/tree/main/tests/pos_simulator"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_pos_sim,node_n_ingestion,node_n_landing toneBlue
class node_n_sync toneAmber
class node_n_bronze_fx,node_n_bronze_weather,node_n_bronze_pos,node_n_bronze_mkt,node_n_bronze_olist,node_n_silver_shared,node_n_silver_sales,node_n_silver_audit,node_n_gold_dims,node_n_gold_facts,node_n_gold_marts toneMint
class node_n_llm toneRose
class node_n_infra,node_n_tests toneIndigo
```

## Components

### Sources

Synthetic and external data producers plus local landed fixtures.

- [POS simulator](https://github.com/satyakiguha007/retail-data-platform/tree/main/pos_simulator): python source generator · Generates synthetic POS transaction logs as partitioned ndjson files. · `pos_simulator`
- [Batch sources](https://github.com/satyakiguha007/retail-data-platform/tree/main/ingestion): source fetchers · FX, weather, marketplace, and Olist source-side generators and fetchers. · `ingestion`
- [Landing data](https://github.com/satyakiguha007/retail-data-platform/tree/main/data/landing/olist): raw fixtures · Representative landed CSV and ndjson artifacts used as inputs and examples. · `data/landing/olist`

### Operations

Human-facing sync and control plane for ADLS upload and scheduling.

- [ADLS console](https://github.com/satyakiguha007/retail-data-platform/tree/main/adls_sync_console): streamlit ops console · Operator console for uploads, scheduling, validation, and sync history. · `adls_sync_console`

### Lakehouse

Databricks-style transformation layers from bronze to gold.

- [FX bronze](https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/01_fx_rates.py): bronze job · Bronze ingestion for FX rate source data. · `transformations/bronze/01_fx_rates.py`
- [Weather bronze](https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/02_weather.py.ipynb): bronze job · Bronze ingestion for weather feeds. · `transformations/bronze/02_weather.py.ipynb`
- [POS bronze](https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/03_pos_rtlog.py): bronze job · Bronze ingestion for POS rtlog files. · `transformations/bronze/03_pos_rtlog.py`
- [Marketplace bronze](https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/04_marketplace.py): bronze job · Bronze ingestion for marketplace feeds. · `transformations/bronze/04_marketplace.py`
- [Olist bronze](https://github.com/satyakiguha007/retail-data-platform/blob/main/transformations/bronze/05_olist.py): bronze job · Bronze ingestion for Olist batch data. · `transformations/bronze/05_olist.py`
- [Silver controls](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver/_shared): shared conformance · Schema gating, quarantine, surrogate keys, and FX helpers. · `transformations/silver/_shared`
- [Sales conformance](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver): canonical sales model · ReSA-style transaction head, item, discount, tender, tax, store-day, and seller normalization. · `transformations/silver`
- [Sales audit](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/silver/audit): reconciliation layer · Rule-based reconciliation and first-class audit error handling. · `transformations/silver/audit`
- [Dimensions](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold): gold marts · Curated dimension tables for date, channel, store, tender, and item. · `transformations/gold`
- [Facts](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold): gold marts · Curated fact tables for sales line, tender, store day, and audit errors. · `transformations/gold`
- [Gold marts](https://github.com/satyakiguha007/retail-data-platform/tree/main/transformations/gold): analytics marts · Business-facing marts such as audit summary, channel P&L, scorecards, and payment mix. · `transformations/gold`

### Serving

Curated outputs and top-layer analytics consumers.

- [LLM layer](https://github.com/satyakiguha007/retail-data-platform/tree/main/llm): consumer analytics · Review enrichment, text-to-SQL, narrative, and client integration scaffolding. · `llm`

### Platform

Infrastructure, delivery, and validation support.

- [Infra](https://github.com/satyakiguha007/retail-data-platform/tree/main/infra): terraform environments · Azure and Databricks environment scaffolding for dev and prod. · `infra`
- [Simulator tests](https://github.com/satyakiguha007/retail-data-platform/tree/main/tests/pos_simulator): pytest suite · Validation for generator, faults, and writer behavior. · `tests/pos_simulator`

## Connections

- POS simulator → ADLS console: local files · POS simulator output is uploaded or synced through the operator console.
- Batch sources → ADLS console: local files · Generated or fetched source artifacts are moved through the ADLS sync boundary.
- Landing data → Olist bronze: batch input · Landing files act as representative batch inputs for Olist ingestion.
- ADLS console → FX bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Weather bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → POS bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Marketplace bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Olist bronze: lands to ADLS · Synced files become bronze-ready source data.
- FX bronze → Silver controls: validate · Bronze FX data is checked and standardized with shared conformance utilities.
- Weather bronze → Silver controls: validate · Bronze weather data is checked and standardized with shared conformance utilities.
- POS bronze → Silver controls: validate · Bronze POS data is checked and standardized with shared conformance utilities.
- Marketplace bronze → Silver controls: validate · Bronze marketplace data is checked and standardized with shared conformance utilities.
- Olist bronze → Silver controls: validate · Bronze Olist data is checked and standardized with shared conformance utilities.
- Silver controls → Sales conformance: conform · Shared gates and keys feed the canonical sales-audit model.
- Sales conformance → Sales audit: audit · Conformed sales data is evaluated by reconciliation rules.
- Sales audit → Facts: error fact · Audit failures are persisted as curated analytical data.
- Sales conformance → Dimensions: dimensions · Conformed entities feed downstream dimensions.
- Sales conformance → Facts: facts · Sales audit model feeds curated fact tables.
- Dimensions → Gold marts: lookup · Dimensions support analytical mart construction.
- Facts → Gold marts: aggregate · Facts are rolled into marts and scorecards.
- Gold marts → LLM layer: curated context · Top-layer LLM features consume curated warehouse outputs.
- Infra → ADLS console: deploy support · Infrastructure scaffolding supports the operational sync boundary.
- Infra → FX bronze: deploy · Infrastructure supports Databricks-style transformation deployment.
- Infra → Sales conformance: deploy · Infrastructure supports Databricks-style transformation deployment.
- Infra → Gold marts: deploy · Infrastructure supports Databricks-style transformation deployment.
- Simulator tests → POS simulator: verify · Tests validate simulator generation and writing behavior.