How praneetsahoo/hcl_tech_hackathon-skyforge works
RetailBank analytics is a batch ETL product: Day 1 and Day 2 raw files are read, validated and standardized, then merged into a MySQL star schema and exposed as KPIs in a password-protected dashboard. The orchestrator owns the batch workflow; the loader handles incremental dimension and fact behavior, while the dashboard can also start a batch. The README documents quarantine, audit/DQ outputs, KPI snapshots, S3 bronze storage and AWS operations. Some subsystem links rely on README documentation because their implementation was not fully excerpted.
Main entry points: run_pipeline.py Batch orchestrator · app.py Analytics dashboard · schema.sql RetailBank MySQL · ingest.py Bronze ingestion
Components 16
Batch pipeline
- Raw batches
data/raw - Batch orchestrator
pipeline/run_pipeline.py - Batch settings
pipeline/config.py - File readers
pipeline/readers.py - Bronze ingestion
pipeline/ingest.py - Entity cleansing
pipeline/cleanse.py - Cleaning rules
pipeline/rules.py
Data storage
- Gold loader
pipeline/load.py - Database access
pipeline/db.py - RetailBank MySQL
sql/schema.sql
Analytics
- KPI views
sql/kpi_views.sql - Analytics dashboard
dashboard/app.py
AWS operations
- S3 storage
- SSM Parameter Store
- CloudWatch monitoring
Other components
- Batch operator
Connections 19
- Batch operator → Batch orchestrator: starts batch
README.md - Raw batches → File readers: supplies files
- Batch orchestrator → Batch settings: gets settings
pipeline/run_pipeline.py - Batch orchestrator → File readers: reads sources
pipeline/run_pipeline.py - Batch orchestrator → Bronze ingestion: uploads files
pipeline/run_pipeline.py - Bronze ingestion → S3 storage: lands bronze
README.md - Batch orchestrator → Entity cleansing: processes entities
pipeline/run_pipeline.py - Entity cleansing → Cleaning rules: applies rules
pipeline/cleanse.py - Batch orchestrator → Gold loader: merges and loads
pipeline/run_pipeline.py - Batch orchestrator → Database access: gets engine
pipeline/run_pipeline.py - Database access → SSM Parameter Store: gets credentials
pipeline/db.py - Gold loader → RetailBank MySQL: writes gold and DQ
README.md - Batch orchestrator → RetailBank MySQL: records run status
pipeline/run_pipeline.py - RetailBank MySQL → KPI views: serves KPI views
README.md - Analytics dashboard → KPI views: queries KPIs
dashboard/app.py - Analytics dashboard → RetailBank MySQL: reads analytics and status
dashboard/app.py - Analytics dashboard → SSM Parameter Store: gets login password
dashboard/app.py - Analytics dashboard → Batch orchestrator: starts batch
dashboard/app.py - Batch orchestrator → CloudWatch monitoring: logs run errors
README.md