How praneetsahoo/hcl_tech_hackathon-team_name- works
RetailBank Analytics turns Day 1/Day 2 raw transaction and master-data files into a cleaned, audited MySQL star schema and 13 KPI views, then presents them in a password-protected dashboard. The principal workflow is batch orchestration → read and cleanse → bronze/quarantine handling → transactional dimension and fact loading → KPI reporting. The graph also preserves data profiling and operational logging. Relationships are limited to calls shown in excerpts or capabilities explicitly documented in the README; unsampled implementation details are not assumed.
Main entry points: run_pipeline.py Batch Orchestrator · app.py Analytics Dashboard · db.py Database Access · readers.py Source Readers
Components 19
Batch Pipeline
- Batch Orchestrator
pipeline/run_pipeline.py - Pipeline Settings
pipeline/config.py - Source Readers
pipeline/readers.py - Bronze Ingest
pipeline/ingest.py - Data Profiler
pipeline/profile_data.py
Data Quality
- Entity Cleansing
pipeline/cleanse.py - Cleaning Rules
pipeline/rules.py - Quarantine Records
pipeline/cleanse.py - DQ and Audit Writes
pipeline/load.py
Analytics Storage
- Gold Loader
pipeline/load.py - Database Access
pipeline/db.py - RetailBank MySQL
- Star Schema
sql/schema.sql - KPI Views
sql/kpi_views.sql
Dashboard and Operations
- Analytics Dashboard
dashboard/app.py - AWS Parameter Store
- CloudWatch Logs
Other components
- Raw Batch Files
data/raw - Pipeline Operator
Connections 23
- Pipeline Operator → Batch Orchestrator: starts batch
README.md - Batch Orchestrator → Pipeline Settings: loads settings
pipeline/run_pipeline.py - Batch Orchestrator → Source Readers: reads files
pipeline/run_pipeline.py - Source Readers → Raw Batch Files: parses
pipeline/readers.py - Batch Orchestrator → Bronze Ingest: uploads raw
pipeline/run_pipeline.py - Bronze Ingest → Pipeline Settings: uses settings
pipeline/ingest.py - Data Profiler → Raw Batch Files: profiles
pipeline/profile_data.py - Data Profiler → Source Readers: salvages JSON
pipeline/profile_data.py - Batch Orchestrator → Entity Cleansing: processes entities
pipeline/run_pipeline.py - Entity Cleansing → Cleaning Rules: applies rules
pipeline/cleanse.py - Batch Orchestrator → Quarantine Records: routes rejects
README.md - Batch Orchestrator → Gold Loader: merges and loads
pipeline/run_pipeline.py - Batch Orchestrator → DQ and Audit Writes: writes scorecard
pipeline/run_pipeline.py - Batch Orchestrator → Database Access: opens connection
pipeline/run_pipeline.py - Database Access → RetailBank MySQL: connects to
pipeline/db.py - Star Schema → RetailBank MySQL: defines tables
README.md - KPI Views → RetailBank MySQL: defines KPI views
README.md - Analytics Dashboard → RetailBank MySQL: queries KPIs
README.md - Analytics Dashboard → Batch Orchestrator: runs batch
dashboard/app.py - Analytics Dashboard → AWS Parameter Store: gets login secret
dashboard/app.py - Database Access → AWS Parameter Store: gets DB secret
pipeline/db.py - Batch Orchestrator → CloudWatch Logs: logs errors
README.md - Analytics Dashboard → CloudWatch Logs: shares pipeline logs
dashboard/app.py