How shashwatnarwal9/idbi_hack works
1. Repository shape
- This is a local-first analytics and scoring platform for bank transactions.
- The core flow is batch data engineering: synthetic/raw data → cleaned transaction layers → customer profiles → validation → prospect scoring → serving/UI.
2. Raw data generation
- src/aayai/datagen.py creates seeded synthetic CSVs with embedded ground truth.
- Those _-prefixed truth columns are a deliberate test/evaluation boundary; they should not leak into transforms, serving, or dashboards.
3. Bronze layer: raw ingestion
- src/aayai/bronze/ingest.py moves raw CSVs into typed Parquet, partitioned by year=YYYY/month=MM.
- This is the first durable data layer and is the bridge from source-like files into the analytics pipeline.
4. Silver layer: transaction normalization
- src/aayai/silver/transform.py and sql/silver_transactions.sql parse narration strings and classify transactions.
- The silver layer is where the repo extracts structure from messy banking text: channel, counterparty, and category.
- src/aayai/silver/evaluate.py compares parsed categories against truth for quality measurement.
5. Gold layer: customer profiles
- src/aayai/gold/build.py and sql/gold_customer_profiles.sql roll transactions up into one row per customer.
- This layer reconstructs monthly income, computes investable surplus, and derives the trust/confidence fields used later in scoring.
- src/aayai/gold/evaluate.py measures income reconstruction accuracy against truth.
6. Validation and confidence gating
- src/aayai/validation/run.py applies data-quality checks, likely backed by Great Expectations as called out in the README.
- The validation output is also used to assign or propagate confidence_band, which becomes a control signal for downstream modeling and orchestration.
7. Modeling layer
- src/aayai/model/train.py trains the prospect model using XGBoost, scikit-learn, and SHAP.
- The model predicts is_good_prospect/prospect score and produces reason codes for explainability.
- A key architectural constraint is that only the training label may touch truth columns; features must not.
8. Serving backend and persisted analytics store
- src/aayai/serving/ holds the database-facing layer: migrations, loading, queries, reviews, and shareable outputs.
- The serving datastore is Postgres, loaded via src/aayai/serving/load.py and managed with migrate.py.
- This is the boundary where curated gold/model outputs become application-ready records for UI consumption.
9. User interfaces
- dashboard/app.py is the Streamlit dashboard described in the README; it reads from the serving layer and presents reconstructed income, spending breakdowns, and SHAP-based prospect explanations.
- frontend/ is a separate React/Vite/TypeScript application with screens for overview, pipeline, validation, customers, upload, deep analysis, and loan assessment.
- The React app uses an API client layer in frontend/src/lib/api.ts and includes mock pipeline data, so it can operate as a richer product UI or demo shell alongside the Streamlit view.
10. API surface
- src/aayai/api/ exposes the application endpoints: customers, overview, pipeline, uploads, loan assessment, and customer PDF output.
- deps.py, config.py, and the endpoint modules suggest a modular HTTP API that sits between the serving database and the UIs.
- This API is the main integration boundary for client apps, while the pipeline code remains offline/batch-oriented.
11. Orchestration and deployment
- airflow/dags/aayai_pipeline.py defines the end-to-end DAG that chains bronze → silver → gold → validation → model.
- The README notes a branch that skips model training when too little trustworthy data is available, so orchestration is not just linear execution but also quality-aware control flow.
- docker-compose.yaml and airflow/Dockerfile support local execution, with Docker used mainly for Airflow and Postgres.
12. SQL as a first-class transformation layer
- The sql/ directory holds the canonical layer definitions for bronze, silver, and gold.
- DuckDB SQL is the core transformation language, with an emphasis on Athena-compatible patterns where possible.
- This keeps the batch pipeline reproducible and portable across local runs.
13. Repository-wide safety and verification
- Tests are organized per subsystem: API, bronze, gold, loan products, model, orchestration, serving, silver, and validation.
- The ground-truth firewall is an important architectural rule: only evaluation code may read truth columns, and serving/UI layers never see them.
- That separation is central to drawing the diagram correctly: data quality, modeling, and presentation are intentionally isolated from the hidden labels used for scoring the synthetic benchmark.
Main entry points: build.py Gold build · aayai_pipeline.py Pipeline DAG · transform.py Silver transform · train.py Model train
Components 20
Batch pipeline
Synthetic data generation and layered analytics transforms.
- Data gen · Creates seeded CSVs with embedded ground truth for evaluation.
src/aayai/datagen.py - Bronze ingest · Converts raw CSVs into partitioned Parquet as the first durable layer.
src/aayai/bronze/ingest.py - Silver transform · Parses narration text and classifies transactions into structured fields.
src/aayai/silver/transform.py - Silver SQL · Canonical SQL definition for normalized transaction records.
sql/silver_transactions.sql - Gold build · Aggregates transactions into one row per customer with profile fields.
src/aayai/gold/build.py - Gold SQL · Canonical SQL definition for customer profiles and derived metrics.
sql/gold_customer_profiles.sql
Quality & model
Validation, evaluation, and prospect scoring logic.
- Validation · Runs quality checks and emits confidence signals for downstream use.
src/aayai/validation/run.py - Silver eval · Measures parsed transaction categories against ground truth.
src/aayai/silver/evaluate.py - Gold eval · Checks reconstructed income against truth columns.
src/aayai/gold/evaluate.py - Model train · Trains the prospect scorer and produces explainable reason codes.
src/aayai/model/train.py
Serving backend
Database persistence, queries, and application-facing outputs.
- Serving DB · Persistent analytics store for curated records and application outputs.
src/aayai/serving/db.py - Migrate · Applies database schema changes for the serving layer.
src/aayai/serving/migrate.py - Load · Loads curated gold and model outputs into Postgres.
src/aayai/serving/load.py - Queries · Database-facing query layer used by APIs and UIs.
src/aayai/serving/queries.py - API · Modular application API for customers, overview, pipeline, uploads, and assessments.
src/aayai/api/main.py
User interfaces
Streamlit dashboard and React/Vite product UI.
- Dashboard · Streamlit dashboard for income, spending, and SHAP-based prospect explanations.
dashboard/app.py - Frontend · React/Vite/TypeScript product UI with multiple workflow screens.
frontend/src/App.tsx - API client · Client layer used by the React app to talk to the backend.
frontend/src/lib/api.ts
Orchestration
Workflow automation and local deployment.
- Pipeline DAG · End-to-end workflow chaining bronze, silver, gold, validation, and model steps.
airflow/dags/aayai_pipeline.py - Compose · Local deployment wiring for Airflow and Postgres.
docker-compose.yaml
Connections 26
- Data gen → Bronze ingest: raw data · Synthetic CSVs feed ingestion into durable storage.
- Bronze ingest → Silver transform: typed transactions · Partitioned transaction data becomes the input for normalization.
- Silver transform → Silver SQL: mirrors · Implementation follows the canonical silver SQL definition.
- Silver transform → Gold build: clean txns · Structured transactions roll up into customer profiles.
- Gold build → Gold SQL: mirrors · Implementation follows the canonical gold SQL definition.
- Silver transform → Silver eval: truth-safe eval · Evaluation reads truth columns; transforms do not.
- Gold build → Gold eval: truth-safe eval · Income accuracy is measured only in evaluation code.
- Gold build → Validation: profile checks · Profile outputs are checked before downstream promotion.
- Validation → Model train: confidence gate · Validation confidence influences whether training proceeds.
- Gold build → Model train: features · Customer features flow into prospect scoring.
- Model train → Load: model outputs · Predictions, scores, and reason codes are published for serving.
- Gold build → Load: gold rows · Curated customer profiles are loaded into Postgres.
- Migrate → Serving DB: schema · Migrations define the persisted serving schema.
- Load → Serving DB: write · Loaded analytics records are stored in the serving database.
- Serving DB → Queries: read · Queries read curated data for application use.
- Queries → API: backend data · HTTP endpoints are backed by serving queries.
- API → API client: json api · React client consumes the application API.
- API → Dashboard: analytics api · Dashboard reads reconstructed profiles and explanations.
- API client → Frontend: client wiring · Shared client helpers power the React screens.
- Pipeline DAG → Bronze ingest: orchestrates · DAG starts the batch pipeline at bronze ingestion.
- Pipeline DAG → Silver transform: orchestrates · DAG continues through silver normalization.
- Pipeline DAG → Gold build: orchestrates · DAG then builds gold customer profiles.
- Pipeline DAG → Validation: orchestrates · Validation is part of the controlled pipeline.
- Pipeline DAG → Model train: conditional · Training can be skipped when confidence is too low.
- Compose → Pipeline DAG: local run · Compose supports local orchestration execution.
- Compose → Serving DB: postgres · Compose also supports the serving datastore.