# praneetsahoo/hcl_tech_hackathon-skyforge architecture

> Architecture diagram of the GitHub repository praneetsahoo/hcl_tech_hackathon-skyforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/praneetsahoo/hcl_tech_hackathon-skyforge
- Repository: https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge

## Overview

RetailBank analytics is a batch ETL product: Day 1 and Day 2 raw files are read, validated and standardized, then merged into a MySQL star schema and exposed as KPIs in a password-protected dashboard. The orchestrator owns the batch workflow; the loader handles incremental dimension and fact behavior, while the dashboard can also start a batch. The README documents quarantine, audit/DQ outputs, KPI snapshots, S3 bronze storage and AWS operations. Some subsystem links rely on README documentation because their implementation was not fully excerpted.

Main entry points: [`pipeline/run_pipeline.py`](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/run_pipeline.py) (Batch orchestrator), [`dashboard/app.py`](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/dashboard/app.py) (Analytics dashboard), [`sql/schema.sql`](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/sql/schema.sql) (RetailBank MySQL), [`pipeline/ingest.py`](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/ingest.py) (Bronze ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Batch pipeline"]
  node_raw_files["Raw batches"]
  node_orchestrator["Batch orchestrator<br/>[run_pipeline.py]"]
  node_settings["Batch settings<br/>[config.py]"]
  node_file_readers["File readers<br/>[readers.py]"]
  node_bronze["Bronze ingestion<br/>[ingest.py]"]
  node_cleaning["Entity cleansing<br/>[cleanse.py]"]
  node_clean_rules["Cleaning rules<br/>[rules.py]"]
end

subgraph group_storage["Data storage"]
  node_gold_loader["Gold loader<br/>[load.py]"]
  node_db_access["Database access<br/>[db.py]"]
  node_mysql[("RetailBank MySQL<br/>[schema.sql]")]
end

subgraph group_analytics["Analytics"]
  node_kpi_views["KPI views<br/>[kpi_views.sql]"]
  node_dashboard["Analytics dashboard<br/>[app.py]"]
end

subgraph group_operations["AWS operations"]
  node_s3[("S3 storage")]
  node_parameter_store[("SSM Parameter Store")]
  node_cloudwatch["CloudWatch monitoring"]
end

node_batch_actor(("Batch operator"))

node_batch_actor -->|"starts batch"| node_orchestrator
node_raw_files -->|"supplies files"| node_file_readers
node_orchestrator -->|"gets settings"| node_settings
node_orchestrator -->|"reads sources"| node_file_readers
node_orchestrator -->|"uploads files"| node_bronze
node_bronze -->|"lands bronze"| node_s3
node_orchestrator -->|"processes entities"| node_cleaning
node_cleaning -->|"applies rules"| node_clean_rules
node_orchestrator -->|"merges and loads"| node_gold_loader
node_orchestrator -->|"gets engine"| node_db_access
node_db_access -->|"gets credentials"| node_parameter_store
node_gold_loader -->|"writes gold and DQ"| node_mysql
node_orchestrator -->|"records run status"| node_mysql
node_mysql -->|"serves KPI views"| node_kpi_views
node_dashboard -->|"queries KPIs"| node_kpi_views
node_dashboard -->|"reads analytics and status"| node_mysql
node_dashboard -->|"gets login password"| node_parameter_store
node_dashboard -->|"starts batch"| node_orchestrator
node_orchestrator -->|"logs run errors"| node_cloudwatch

click node_raw_files "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/tree/main/data/raw"
click node_orchestrator "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/run_pipeline.py"
click node_settings "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/config.py"
click node_file_readers "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/readers.py"
click node_bronze "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/ingest.py"
click node_cleaning "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/cleanse.py"
click node_clean_rules "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/rules.py"
click node_gold_loader "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/load.py"
click node_db_access "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/db.py"
click node_mysql "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/sql/schema.sql"
click node_kpi_views "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/sql/kpi_views.sql"
click node_dashboard "https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_files,node_orchestrator,node_settings,node_file_readers,node_bronze,node_cleaning,node_clean_rules toneBlue
class node_gold_loader,node_db_access,node_mysql toneAmber
class node_kpi_views,node_dashboard toneMint
class node_s3,node_parameter_store,node_cloudwatch toneRose
class node_batch_actor toneIndigo
```

## Components

### Batch pipeline

- [Raw batches](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/tree/main/data/raw): `data/raw`
- [Batch orchestrator](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/run_pipeline.py): `pipeline/run_pipeline.py`
- [Batch settings](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/config.py): `pipeline/config.py`
- [File readers](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/readers.py): `pipeline/readers.py`
- [Bronze ingestion](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/ingest.py): `pipeline/ingest.py`
- [Entity cleansing](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/cleanse.py): `pipeline/cleanse.py`
- [Cleaning rules](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/rules.py): `pipeline/rules.py`

### Data storage

- [Gold loader](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/load.py): `pipeline/load.py`
- [Database access](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/pipeline/db.py): `pipeline/db.py`
- [RetailBank MySQL](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/sql/schema.sql): `sql/schema.sql`

### Analytics

- [KPI views](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/sql/kpi_views.sql): `sql/kpi_views.sql`
- [Analytics dashboard](https://github.com/praneetsahoo/hcl_tech_hackathon-skyforge/blob/main/dashboard/app.py): `dashboard/app.py`

### AWS operations

- **S3 storage**
- **SSM Parameter Store**
- **CloudWatch monitoring**

### Other components

- **Batch operator**

## Connections

- Batch operator → Batch orchestrator: starts batch (evidence: `README.md`)
- Raw batches → File readers: supplies files
- Batch orchestrator → Batch settings: gets settings (evidence: `pipeline/run_pipeline.py`)
- Batch orchestrator → File readers: reads sources (evidence: `pipeline/run_pipeline.py`)
- Batch orchestrator → Bronze ingestion: uploads files (evidence: `pipeline/run_pipeline.py`)
- Bronze ingestion → S3 storage: lands bronze (evidence: `README.md`)
- Batch orchestrator → Entity cleansing: processes entities (evidence: `pipeline/run_pipeline.py`)
- Entity cleansing → Cleaning rules: applies rules (evidence: `pipeline/cleanse.py`)
- Batch orchestrator → Gold loader: merges and loads (evidence: `pipeline/run_pipeline.py`)
- Batch orchestrator → Database access: gets engine (evidence: `pipeline/run_pipeline.py`)
- Database access → SSM Parameter Store: gets credentials (evidence: `pipeline/db.py`)
- Gold loader → RetailBank MySQL: writes gold and DQ (evidence: `README.md`)
- Batch orchestrator → RetailBank MySQL: records run status (evidence: `pipeline/run_pipeline.py`)
- RetailBank MySQL → KPI views: serves KPI views (evidence: `README.md`)
- Analytics dashboard → KPI views: queries KPIs (evidence: `dashboard/app.py`)
- Analytics dashboard → RetailBank MySQL: reads analytics and status (evidence: `dashboard/app.py`)
- Analytics dashboard → SSM Parameter Store: gets login password (evidence: `dashboard/app.py`)
- Analytics dashboard → Batch orchestrator: starts batch (evidence: `dashboard/app.py`)
- Batch orchestrator → CloudWatch monitoring: logs run errors (evidence: `README.md`)
