How hsai-rohit/real-time-ecommerce-data-platform works
This is a batch-oriented e-commerce data ingestion and validation platform, with local CSV, SQLite, and mock-API sources plus ADLS Gen2 and Databricks integrations. The principal workflow is to run a source-specific ingestion command, land raw data, validate its contracts and relationships, then optionally upload it to cloud storage. The excerpts show several runner-to-ingester calls and validation behavior; unsampled orchestration and validation runners are represented without asserted call links.
Main entry points: data Raw Data Landing · api_ingestion.py API Landing · csv_validation.py CSV Contract Checks · sqlite_validation.py Orders & Payments Checks
Components 19
Source Ingestion
- CSV Ingestion Runner
scripts/run_csv_ingestion.py - CSV Landing
ingestion/csv_ingestion.py - SQLite Ingestion Runner
scripts/run_sqlite_ingestion.py - SQLite Extraction
ingestion/sqlite_ingestion.py - API Ingestion Runner
scripts/run_api_ingestion.py - API Landing
ingestion/api_ingestion.py - Raw Data Landing
data
Data Validation
- CSV Contract Checks
validation/csv_validation.py - Orders & Payments Checks
validation/sqlite_validation.py - Events API Checks
validation/api_validation.py
Cloud Storage
- ADLS Upload
ingestion/adls_upload.py - ADLS Download
ingestion/adls_download.py - Databricks Volume Upload
ingestion/databricks_volume_upload.py
Local Sources
- Customer & Product CSV
data/source/csv - Orders & Payments DB
data/source/sqlite - Mock Events API
scripts/mock_events_api.py
Other components
- Operator
- Azure ADLS Gen2
- Databricks Volume
Connections 21
- Operator → CSV Ingestion Runner: runs
- Operator → SQLite Ingestion Runner: runs
- Operator → API Ingestion Runner: runs
- CSV Ingestion Runner → CSV Landing: calls
scripts/run_csv_ingestion.py - SQLite Ingestion Runner → SQLite Extraction: calls
scripts/run_sqlite_ingestion.py - API Ingestion Runner → API Landing: calls
scripts/run_api_ingestion.py - CSV Landing → Raw Data Landing: writes
ingestion/csv_ingestion.py - SQLite Extraction → Raw Data Landing: writes
ingestion/sqlite_ingestion.py - API Landing → Mock Events API: fetches JSON
ingestion/api_ingestion.py - API Landing → Raw Data Landing: writes
ingestion/api_ingestion.py - CSV Contract Checks → Raw Data Landing: checks CSV
- Orders & Payments Checks → Raw Data Landing: checks CSV
- Events API Checks → Raw Data Landing: checks JSON
- Events API Checks → CSV Contract Checks: uses result type
validation/api_validation.py - Orders & Payments Checks → CSV Contract Checks: uses helpers
validation/sqlite_validation.py - Operator → ADLS Upload: runs upload
- ADLS Upload → Azure ADLS Gen2: uploads files
ingestion/adls_upload.py - Azure ADLS Gen2 → ADLS Download: downloads prefix
ingestion/adls_download.py - Operator → ADLS Download: runs download
- Operator → Databricks Volume Upload: runs upload
- Databricks Volume Upload → Databricks Volume: uploads files
ingestion/databricks_volume_upload.py