How satyakiguha007/retail-data-platform works
1. Overall shape
- This repo is a retail data platform on Azure + Databricks, organized as a staged pipeline: source simulation/collection → bronze ingestion → silver conformance and sales audit → gold marts/metrics → optional LLM analytics.
- The main architectural boundary is between data-producing code in Python and lakehouse transformation code in Databricks-style scripts/notebooks.
2. Source systems and generators
- pos_simulator/ is the primary synthetic operational source: it produces point-of-sale transaction logs as partitioned ndjson files by store/date/hour.
- ingestion/ contains additional source-side generators/fetchers for FX rates, weather, marketplace feeds, and Olist reference/e-commerce data.
- These sources are independent of the lakehouse logic; they create raw artifacts that later feed ingestion jobs.
3. Raw data layout
- The repository includes representative landed data under output/ and data/landing/.
- output/mkt_feed/.../feed.ndjson and output/pos_rtlog/.../rtlog.ndjson show the expected raw partitioning pattern.
- data/landing/olist/ holds CSV inputs for the Olist dataset, indicating the platform supports both file-based batch inputs and generated event-style feeds.
4. Operator/control plane
- adls_sync_console/ is a separate Streamlit-based console used to sync data to ADLS and manage uploads/scheduling/statistics.
- Its core modules (adls.py, config.py, scheduler.py, state.py, sync.py, validate.py) suggest it acts as the human-facing operational boundary for moving local files into cloud storage and tracking sync history (data/history.json).
5. Bronze ingestion layer
- transformations/bronze/ contains the first Databricks processing stage, with one job per source: FX rates, weather, POS RT logs, marketplace, Olist, stores, and SA store data.
- This layer is responsible for getting raw source files into a standardized bronze representation, likely with source-specific validation and minimal reshaping.
- The presence of notebooks and Python scripts suggests mixed interactive and job-oriented execution.
6. Cross-cutting ingestion safeguards
- transformations/silver/_shared/ includes reusable controls such as schema_gate.py, quarantine.py, fx_helpers.py, and surrogate_keys.py.
- These indicate the platform enforces schema/quality gates and separates rejected records from clean records before conformance.
- This is an important boundary for the diagram: bronze can accept messy source data; silver only accepts validated/conformed data.
7. Silver conformance model
- transformations/silver/ is centered on an Oracle ReSA-style canonical sales model.
- The folder structure mirrors the business entities: transaction header, item, discount, tender, tax, inter-company tax, store-day, and seller data.
- Separate source adapters exist for marketplace, POS, Olist, and seller/store-related feeds, showing that multiple channels are normalized into one sales-audit schema.
8. Sales audit subsystem
- transformations/silver/audit/ is a distinct reconciliation layer with a rule framework and 18 specific rules under rules/.
- The rules cover count, total, foreign-key, sign, validation, outlier, and business-date consistency checks, plus currency alignment.
- sa_error_writer.py and the shared rule framework imply audit failures are persisted as first-class data, not just logs.
9. Gold analytics layer
- transformations/gold/ builds downstream dimensions, facts, and marts from the conformed/audited silver data.
- Key outputs include dim_date, dim_channel, dim_store, dim_tender, dim_item, and facts such as fact_sales_line, fact_tender, fact_store_day, plus marts for audit summary, channel P&L, store-day scorecards, and payment mix.
- Shared helpers like cdf_reader.py, dim_lookup.py, and watermark.py suggest incremental processing and dimension lookup patterns.
10. LLM layer
- llm/ is a separate top-layer capability with placeholders for review enrichment, text-to-SQL, weekly narrative, and client integrations.
- Its location shows it consumes curated warehouse outputs rather than raw feeds, so it belongs above gold in the architecture.
- The repo tree suggests this layer is scaffolded but not yet populated with implementation.
11. Infrastructure and delivery
- infra/ contains Terraform environment scaffolding for dev and prod, so cloud resources are managed as code.
- The README also calls out GitHub Actions CI/CD and Databricks Asset Bundles, implying deployment automation into Databricks and Azure storage/runtime resources.
- scripts/test_adls_connection.py provides a small operational check for Azure storage connectivity.
12. Testing and sample fixtures
- tests/pos_simulator/ validates the simulator’s generator, faults, and writer behavior.
- sample_data/ directories under the simulator and ingestion sources provide reproducible fixtures for local development and smoke tests.
- This supports a workflow where local generation can be verified before data is promoted into the lakehouse pipeline.
13. Diagram boundaries to preserve
- Treat the repo as four layers: source generation/control plane, raw landing/bronze, conformed silver with audit, and serving gold + LLM analytics.
- The critical data-flow arrows are:
- source generators → landed files/ADLS
- bronze jobs → validated/conformed silver tables
- silver audit rules → audit error outputs
- silver/gold outputs → analytics marts and LLM consumers
- The main operational boundary is the Streamlit sync console; the main analytical boundary is the Databricks lakehouse.
Main entry points: adls_sync_console ADLS console · _shared Silver controls · silver Sales conformance · infra Infra
Components 18
Sources
Synthetic and external data producers plus local landed fixtures.
- POS simulator · Generates synthetic POS transaction logs as partitioned ndjson files.
pos_simulator - Batch sources · FX, weather, marketplace, and Olist source-side generators and fetchers.
ingestion - Landing data · Representative landed CSV and ndjson artifacts used as inputs and examples.
data/landing/olist
Operations
Human-facing sync and control plane for ADLS upload and scheduling.
- ADLS console · Operator console for uploads, scheduling, validation, and sync history.
adls_sync_console
Lakehouse
Databricks-style transformation layers from bronze to gold.
- FX bronze · Bronze ingestion for FX rate source data.
transformations/bronze/01_fx_rates.py - Weather bronze · Bronze ingestion for weather feeds.
transformations/bronze/02_weather.py.ipynb - POS bronze · Bronze ingestion for POS rtlog files.
transformations/bronze/03_pos_rtlog.py - Marketplace bronze · Bronze ingestion for marketplace feeds.
transformations/bronze/04_marketplace.py - Olist bronze · Bronze ingestion for Olist batch data.
transformations/bronze/05_olist.py - Silver controls · Schema gating, quarantine, surrogate keys, and FX helpers.
transformations/silver/_shared - Sales conformance · ReSA-style transaction head, item, discount, tender, tax, store-day, and seller normalization.
transformations/silver - Sales audit · Rule-based reconciliation and first-class audit error handling.
transformations/silver/audit - Dimensions · Curated dimension tables for date, channel, store, tender, and item.
transformations/gold - Facts · Curated fact tables for sales line, tender, store day, and audit errors.
transformations/gold - Gold marts · Business-facing marts such as audit summary, channel P&L, scorecards, and payment mix.
transformations/gold
Serving
Curated outputs and top-layer analytics consumers.
- LLM layer · Review enrichment, text-to-SQL, narrative, and client integration scaffolding.
llm
Platform
Infrastructure, delivery, and validation support.
- Infra · Azure and Databricks environment scaffolding for dev and prod.
infra - Simulator tests · Validation for generator, faults, and writer behavior.
tests/pos_simulator
Connections 26
- POS simulator → ADLS console: local files · POS simulator output is uploaded or synced through the operator console.
- Batch sources → ADLS console: local files · Generated or fetched source artifacts are moved through the ADLS sync boundary.
- Landing data → Olist bronze: batch input · Landing files act as representative batch inputs for Olist ingestion.
- ADLS console → FX bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Weather bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → POS bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Marketplace bronze: lands to ADLS · Synced files become bronze-ready source data.
- ADLS console → Olist bronze: lands to ADLS · Synced files become bronze-ready source data.
- FX bronze → Silver controls: validate · Bronze FX data is checked and standardized with shared conformance utilities.
- Weather bronze → Silver controls: validate · Bronze weather data is checked and standardized with shared conformance utilities.
- POS bronze → Silver controls: validate · Bronze POS data is checked and standardized with shared conformance utilities.
- Marketplace bronze → Silver controls: validate · Bronze marketplace data is checked and standardized with shared conformance utilities.
- Olist bronze → Silver controls: validate · Bronze Olist data is checked and standardized with shared conformance utilities.
- Silver controls → Sales conformance: conform · Shared gates and keys feed the canonical sales-audit model.
- Sales conformance → Sales audit: audit · Conformed sales data is evaluated by reconciliation rules.
- Sales audit → Facts: error fact · Audit failures are persisted as curated analytical data.
- Sales conformance → Dimensions: dimensions · Conformed entities feed downstream dimensions.
- Sales conformance → Facts: facts · Sales audit model feeds curated fact tables.
- Dimensions → Gold marts: lookup · Dimensions support analytical mart construction.
- Facts → Gold marts: aggregate · Facts are rolled into marts and scorecards.
- Gold marts → LLM layer: curated context · Top-layer LLM features consume curated warehouse outputs.
- Infra → ADLS console: deploy support · Infrastructure scaffolding supports the operational sync boundary.
- Infra → FX bronze: deploy · Infrastructure supports Databricks-style transformation deployment.
- Infra → Sales conformance: deploy · Infrastructure supports Databricks-style transformation deployment.
- Infra → Gold marts: deploy · Infrastructure supports Databricks-style transformation deployment.
- Simulator tests → POS simulator: verify · Tests validate simulator generation and writing behavior.