# shashwatnarwal9/idbi_hack architecture

> Architecture diagram of the GitHub repository shashwatnarwal9/idbi_hack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shashwatnarwal9/idbi_hack
- Repository: https://github.com/shashwatnarwal9/idbi_hack

## Overview

1. **Repository shape**
   - This is a local-first analytics and scoring platform for bank transactions.
   - The core flow is batch data engineering: synthetic/raw data → cleaned transaction layers → customer profiles → validation → prospect scoring → serving/UI.

2. **Raw data generation**
   - `src/aayai/datagen.py` creates seeded synthetic CSVs with embedded ground truth.
   - Those `_`-prefixed truth columns are a deliberate test/evaluation boundary; they should not leak into transforms, serving, or dashboards.

3. **Bronze layer: raw ingestion**
   - `src/aayai/bronze/ingest.py` moves raw CSVs into typed Parquet, partitioned by `year=YYYY/month=MM`.
   - This is the first durable data layer and is the bridge from source-like files into the analytics pipeline.

4. **Silver layer: transaction normalization**
   - `src/aayai/silver/transform.py` and `sql/silver_transactions.sql` parse narration strings and classify transactions.
   - The silver layer is where the repo extracts structure from messy banking text: channel, counterparty, and category.
   - `src/aayai/silver/evaluate.py` compares parsed categories against truth for quality measurement.

5. **Gold layer: customer profiles**
   - `src/aayai/gold/build.py` and `sql/gold_customer_profiles.sql` roll transactions up into one row per customer.
   - This layer reconstructs monthly income, computes investable surplus, and derives the trust/confidence fields used later in scoring.
   - `src/aayai/gold/evaluate.py` measures income reconstruction accuracy against truth.

6. **Validation and confidence gating**
   - `src/aayai/validation/run.py` applies data-quality checks, likely backed by Great Expectations as called out in the README.
   - The validation output is also used to assign or propagate `confidence_band`, which becomes a control signal for downstream modeling and orchestration.

7. **Modeling layer**
   - `src/aayai/model/train.py` trains the prospect model using XGBoost, scikit-learn, and SHAP.
   - The model predicts `is_good_prospect`/prospect score and produces reason codes for explainability.
   - A key architectural constraint is that only the training label may touch truth columns; features must not.

8. **Serving backend and persisted analytics store**
   - `src/aayai/serving/` holds the database-facing layer: migrations, loading, queries, reviews, and shareable outputs.
   - The serving datastore is Postgres, loaded via `src/aayai/serving/load.py` and managed with `migrate.py`.
   - This is the boundary where curated gold/model outputs become application-ready records for UI consumption.

9. **User interfaces**
   - `dashboard/app.py` is the Streamlit dashboard described in the README; it reads from the serving layer and presents reconstructed income, spending breakdowns, and SHAP-based prospect explanations.
   - `frontend/` is a separate React/Vite/TypeScript application with screens for overview, pipeline, validation, customers, upload, deep analysis, and loan assessment.
   - The React app uses an API client layer in `frontend/src/lib/api.ts` and includes mock pipeline data, so it can operate as a richer product UI or demo shell alongside the Streamlit view.

10. **API surface**
   - `src/aayai/api/` exposes the application endpoints: customers, overview, pipeline, uploads, loan assessment, and customer PDF output.
   - `deps.py`, `config.py`, and the endpoint modules suggest a modular HTTP API that sits between the serving database and the UIs.
   - This API is the main integration boundary for client apps, while the pipeline code remains offline/batch-oriented.

11. **Orchestration and deployment**
   - `airflow/dags/aayai_pipeline.py` defines the end-to-end DAG that chains bronze → silver → gold → validation → model.
   - The README notes a branch that skips model training when too little trustworthy data is available, so orchestration is not just linear execution but also quality-aware control flow.
   - `docker-compose.yaml` and `airflow/Dockerfile` support local execution, with Docker used mainly for Airflow and Postgres.

12. **SQL as a first-class transformation layer**
   - The `sql/` directory holds the canonical layer definitions for bronze, silver, and gold.
   - DuckDB SQL is the core transformation language, with an emphasis on Athena-compatible patterns where possible.
   - This keeps the batch pipeline reproducible and portable across local runs.

13. **Repository-wide safety and verification**
   - Tests are organized per subsystem: API, bronze, gold, loan products, model, orchestration, serving, silver, and validation.
   - The `ground-truth firewall` is an important architectural rule: only evaluation code may read truth columns, and serving/UI layers never see them.
   - That separation is central to drawing the diagram correctly: data quality, modeling, and presentation are intentionally isolated from the hidden labels used for scoring the synthetic benchmark.

Main entry points: [`src/aayai/gold/build.py`](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/gold/build.py) (Gold build), [`airflow/dags/aayai_pipeline.py`](https://github.com/shashwatnarwal9/idbi_hack/blob/main/airflow/dags/aayai_pipeline.py) (Pipeline DAG), [`src/aayai/silver/transform.py`](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/silver/transform.py) (Silver transform), [`src/aayai/model/train.py`](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/model/train.py) (Model train)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Batch pipeline"]
  node_datagen["Data gen<br/>synthetic data<br/>[datagen.py]"]
  node_bronze_ingest[("Bronze ingest<br/>raw->parquet<br/>[ingest.py]")]
  node_silver_transform["Silver transform<br/>txn normalization<br/>[transform.py]"]
  node_silver_sql["Silver SQL<br/>layer sql"]
  node_gold_build[("Gold build<br/>customer rollup<br/>[build.py]")]
  node_gold_sql["Gold SQL<br/>layer sql"]
end

subgraph group_quality["Quality & model"]
  node_validation_run["Validation<br/>data quality<br/>[run.py]"]
  node_silver_evaluate["Silver eval<br/>classification eval<br/>[evaluate.py]"]
  node_gold_evaluate["Gold eval<br/>income eval<br/>[evaluate.py]"]
  node_model_train["Model train<br/>prospect model<br/>[train.py]"]
end

subgraph group_serving["Serving backend"]
  node_serving_db[("Serving DB<br/>postgres store<br/>[db.py]")]
  node_serving_migrate["Migrate<br/>schema mgmt<br/>[migrate.py]"]
  node_serving_load["Load<br/>publish curated<br/>[load.py]"]
  node_serving_queries["Queries<br/>read models<br/>[queries.py]"]
  node_api_main["API<br/>http surface<br/>[main.py]"]
end

subgraph group_ui["User interfaces"]
  node_dashboard_app["Dashboard<br/>streamlit ui<br/>[app.py]"]
  node_frontend_app["Frontend<br/>react ui<br/>[App.tsx]"]
  node_frontend_api["API client<br/>ui client<br/>[api.ts]"]
end

subgraph group_ops["Orchestration"]
  node_airflow_dag["Pipeline DAG<br/>orchestration<br/>[aayai_pipeline.py]"]
  node_compose["Compose<br/>local runtime"]
end

node_datagen -->|"raw data"| node_bronze_ingest
node_bronze_ingest -->|"typed transactions"| node_silver_transform
node_silver_transform -.->|"mirrors"| node_silver_sql
node_silver_transform -->|"clean txns"| node_gold_build
node_gold_build -.->|"mirrors"| node_gold_sql
node_silver_transform -.->|"truth-safe eval"| node_silver_evaluate
node_gold_build -.->|"truth-safe eval"| node_gold_evaluate
node_gold_build -->|"profile checks"| node_validation_run
node_validation_run -->|"confidence gate"| node_model_train
node_gold_build -->|"features"| node_model_train
node_model_train -->|"model outputs"| node_serving_load
node_gold_build -->|"gold rows"| node_serving_load
node_serving_migrate -->|"schema"| node_serving_db
node_serving_load -->|"write"| node_serving_db
node_serving_db -->|"read"| node_serving_queries
node_serving_queries -->|"backend data"| node_api_main
node_api_main -->|"json api"| node_frontend_api
node_api_main -->|"analytics api"| node_dashboard_app
node_frontend_api -->|"client wiring"| node_frontend_app
node_airflow_dag -->|"orchestrates"| node_bronze_ingest
node_airflow_dag -->|"orchestrates"| node_silver_transform
node_airflow_dag -->|"orchestrates"| node_gold_build
node_airflow_dag -->|"orchestrates"| node_validation_run
node_airflow_dag -.->|"conditional"| node_model_train
node_compose -.->|"local run"| node_airflow_dag
node_compose -.->|"postgres"| node_serving_db

click node_datagen "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/datagen.py"
click node_bronze_ingest "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/bronze/ingest.py"
click node_silver_transform "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/silver/transform.py"
click node_silver_sql "https://github.com/shashwatnarwal9/idbi_hack/blob/main/sql/silver_transactions.sql"
click node_gold_build "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/gold/build.py"
click node_gold_sql "https://github.com/shashwatnarwal9/idbi_hack/blob/main/sql/gold_customer_profiles.sql"
click node_validation_run "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/validation/run.py"
click node_silver_evaluate "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/silver/evaluate.py"
click node_gold_evaluate "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/gold/evaluate.py"
click node_model_train "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/model/train.py"
click node_serving_db "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/db.py"
click node_serving_migrate "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/migrate.py"
click node_serving_load "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/load.py"
click node_serving_queries "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/queries.py"
click node_api_main "https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/api/main.py"
click node_dashboard_app "https://github.com/shashwatnarwal9/idbi_hack/blob/main/dashboard/app.py"
click node_frontend_app "https://github.com/shashwatnarwal9/idbi_hack/blob/main/frontend/src/App.tsx"
click node_frontend_api "https://github.com/shashwatnarwal9/idbi_hack/blob/main/frontend/src/lib/api.ts"
click node_airflow_dag "https://github.com/shashwatnarwal9/idbi_hack/blob/main/airflow/dags/aayai_pipeline.py"
click node_compose "https://github.com/shashwatnarwal9/idbi_hack/blob/main/docker-compose.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_datagen,node_bronze_ingest,node_silver_transform,node_silver_sql,node_gold_build,node_gold_sql toneBlue
class node_validation_run,node_silver_evaluate,node_gold_evaluate,node_model_train toneAmber
class node_serving_db,node_serving_migrate,node_serving_load,node_serving_queries,node_api_main toneMint
class node_dashboard_app,node_frontend_app,node_frontend_api toneRose
class node_airflow_dag,node_compose toneIndigo
```

## Components

### Batch pipeline

Synthetic data generation and layered analytics transforms.

- [Data gen](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/datagen.py): synthetic data · Creates seeded CSVs with embedded ground truth for evaluation. · `src/aayai/datagen.py`
- [Bronze ingest](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/bronze/ingest.py): raw->parquet · Converts raw CSVs into partitioned Parquet as the first durable layer. · `src/aayai/bronze/ingest.py`
- [Silver transform](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/silver/transform.py): txn normalization · Parses narration text and classifies transactions into structured fields. · `src/aayai/silver/transform.py`
- [Silver SQL](https://github.com/shashwatnarwal9/idbi_hack/blob/main/sql/silver_transactions.sql): layer sql · Canonical SQL definition for normalized transaction records. · `sql/silver_transactions.sql`
- [Gold build](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/gold/build.py): customer rollup · Aggregates transactions into one row per customer with profile fields. · `src/aayai/gold/build.py`
- [Gold SQL](https://github.com/shashwatnarwal9/idbi_hack/blob/main/sql/gold_customer_profiles.sql): layer sql · Canonical SQL definition for customer profiles and derived metrics. · `sql/gold_customer_profiles.sql`

### Quality & model

Validation, evaluation, and prospect scoring logic.

- [Validation](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/validation/run.py): data quality · Runs quality checks and emits confidence signals for downstream use. · `src/aayai/validation/run.py`
- [Silver eval](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/silver/evaluate.py): classification eval · Measures parsed transaction categories against ground truth. · `src/aayai/silver/evaluate.py`
- [Gold eval](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/gold/evaluate.py): income eval · Checks reconstructed income against truth columns. · `src/aayai/gold/evaluate.py`
- [Model train](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/model/train.py): prospect model · Trains the prospect scorer and produces explainable reason codes. · `src/aayai/model/train.py`

### Serving backend

Database persistence, queries, and application-facing outputs.

- [Serving DB](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/db.py): postgres store · Persistent analytics store for curated records and application outputs. · `src/aayai/serving/db.py`
- [Migrate](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/migrate.py): schema mgmt · Applies database schema changes for the serving layer. · `src/aayai/serving/migrate.py`
- [Load](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/load.py): publish curated · Loads curated gold and model outputs into Postgres. · `src/aayai/serving/load.py`
- [Queries](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/serving/queries.py): read models · Database-facing query layer used by APIs and UIs. · `src/aayai/serving/queries.py`
- [API](https://github.com/shashwatnarwal9/idbi_hack/blob/main/src/aayai/api/main.py): http surface · Modular application API for customers, overview, pipeline, uploads, and assessments. · `src/aayai/api/main.py`

### User interfaces

Streamlit dashboard and React/Vite product UI.

- [Dashboard](https://github.com/shashwatnarwal9/idbi_hack/blob/main/dashboard/app.py): streamlit ui · Streamlit dashboard for income, spending, and SHAP-based prospect explanations. · `dashboard/app.py`
- [Frontend](https://github.com/shashwatnarwal9/idbi_hack/blob/main/frontend/src/App.tsx): react ui · React/Vite/TypeScript product UI with multiple workflow screens. · `frontend/src/App.tsx`
- [API client](https://github.com/shashwatnarwal9/idbi_hack/blob/main/frontend/src/lib/api.ts): ui client · Client layer used by the React app to talk to the backend. · `frontend/src/lib/api.ts`

### Orchestration

Workflow automation and local deployment.

- [Pipeline DAG](https://github.com/shashwatnarwal9/idbi_hack/blob/main/airflow/dags/aayai_pipeline.py): orchestration · End-to-end workflow chaining bronze, silver, gold, validation, and model steps. · `airflow/dags/aayai_pipeline.py`
- [Compose](https://github.com/shashwatnarwal9/idbi_hack/blob/main/docker-compose.yaml): local runtime · Local deployment wiring for Airflow and Postgres. · `docker-compose.yaml`

## Connections

- Data gen → Bronze ingest: raw data · Synthetic CSVs feed ingestion into durable storage.
- Bronze ingest → Silver transform: typed transactions · Partitioned transaction data becomes the input for normalization.
- Silver transform → Silver SQL: mirrors · Implementation follows the canonical silver SQL definition.
- Silver transform → Gold build: clean txns · Structured transactions roll up into customer profiles.
- Gold build → Gold SQL: mirrors · Implementation follows the canonical gold SQL definition.
- Silver transform → Silver eval: truth-safe eval · Evaluation reads truth columns; transforms do not.
- Gold build → Gold eval: truth-safe eval · Income accuracy is measured only in evaluation code.
- Gold build → Validation: profile checks · Profile outputs are checked before downstream promotion.
- Validation → Model train: confidence gate · Validation confidence influences whether training proceeds.
- Gold build → Model train: features · Customer features flow into prospect scoring.
- Model train → Load: model outputs · Predictions, scores, and reason codes are published for serving.
- Gold build → Load: gold rows · Curated customer profiles are loaded into Postgres.
- Migrate → Serving DB: schema · Migrations define the persisted serving schema.
- Load → Serving DB: write · Loaded analytics records are stored in the serving database.
- Serving DB → Queries: read · Queries read curated data for application use.
- Queries → API: backend data · HTTP endpoints are backed by serving queries.
- API → API client: json api · React client consumes the application API.
- API → Dashboard: analytics api · Dashboard reads reconstructed profiles and explanations.
- API client → Frontend: client wiring · Shared client helpers power the React screens.
- Pipeline DAG → Bronze ingest: orchestrates · DAG starts the batch pipeline at bronze ingestion.
- Pipeline DAG → Silver transform: orchestrates · DAG continues through silver normalization.
- Pipeline DAG → Gold build: orchestrates · DAG then builds gold customer profiles.
- Pipeline DAG → Validation: orchestrates · Validation is part of the controlled pipeline.
- Pipeline DAG → Model train: conditional · Training can be skipped when confidence is too low.
- Compose → Pipeline DAG: local run · Compose supports local orchestration execution.
- Compose → Serving DB: postgres · Compose also supports the serving datastore.
