# m03jam14/insightengine architecture

> Architecture diagram of the GitHub repository m03jam14/insightengine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/m03jam14/insightengine
- Repository: https://github.com/m03jam14/insightengine

## Overview

##### Project Type and Purpose
- This is a data‐driven analytics dashboard built on Tableau, focused on visualizing Canadian variable mortgage rates (2015–2024).
- It combines ETL pipelines, a data warehouse, statistical analysis modules, and an interactive Tableau front end for financial and policy stakeholders.

##### Main Components
a. External Data Sources
- Bank of Canada rate publications (API or flat files)
- Housing affordability indices, economic indicators, quarterly financial data
b. Data Processing & Engineering
- Python ETL scripts (ingest, clean, normalize)
- SQL jobs for staging, transformation, and loading into the warehouse
- Automated scheduler (cron, Airflow, or similar)
c. Data Warehouse
- Staging schema for raw data
- Fact tables for processed rate series and calculated indices
- Dimension tables (dates, rate environments, geographies)
d. Advanced Analytics Layer
- Statistical modules (volatility calculations, time‐series decomposition, correlation analysis)
- Stored procedures or Python/R scripts invoked by SQL or scheduler
e. Presentation Layer
- Tableau Desktop workbooks (.twbx)
- Tableau Server for publication, user access, and scheduled data refresh
- Interactive front end: dashboards, parameter controls, filters
f. Users & External Interfaces
- Financial institutions, policy analysts accessing via web browsers
- API or export endpoints for downstream reporting tools

##### Relationships and Data Flow
1. External Data Sources → Python ETL (extract, validate)
2. Python ETL → Staging Schema in Data Warehouse
3. SQL Transformations & Analytics Modules → Fact/Dimension Tables
4. Data Warehouse → Tableau Server Data Extracts (scheduled refresh)
5. Tableau Server → End‐user Dashboards (interactive analysis)

##### Architectural Patterns & Design Principles
- Modular ETL (separation of extraction, transformation, loading)
- ELT vs. in‐database analytics for performance (push analytics into SQL where possible)
- Layered architecture: raw → curated → presentation
- Scalable scheduling: handle daily or real‐time updates
- Responsive & caching strategies on Tableau Server for sub‐2‐second load times

##### Technologies & Frameworks
- Python (ETL orchestration, validation scripts)
- SQL (data modeling, analytic views, stored procedures)
- Data warehouse (PostgreSQL, Snowflake, or equivalent)
- Tableau Desktop / Server (2021.1+)
- Scheduler (Airflow, cron jobs, or Tableau scheduling)
- Version control (Git for code, workbook versioning)

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% External Data Sources
  subgraph "External Data Sources"
    BOC["Bank of Canada Rates\n(API or files)"]:::external
    Housing["Housing Affordability &\nEconomic Indicators"]:::external
  end

  %% Data Processing & Engineering
  subgraph "Data Processing & Engineering"
    Scheduler["Scheduler\n(Airflow / Cron)"]:::compute
    ETL["ETL: Python Scripts\n(Ingest, Clean, Normalize)"]:::compute
    SQLJobs["SQL Jobs\n(Staging & Transform)"]:::compute
  end

  %% Data Warehouse
  subgraph "Data Warehouse"
    Staging["Schema: Staging"]:::storage
    Fact["Fact Tables\n(MortgageRate_Fact)"]:::storage
    Dim["Dimension Tables\n(Dates, Geographies)"]:::storage
  end

  %% Advanced Analytics Layer
  subgraph "Advanced Analytics Layer"
    Stats["Statistical Modules\n(Volatility, Decomposition)"]:::compute
    Proc["Stored Procs / Python-R\n(In-DB Analytics)"]:::compute
  end

  %% Presentation Layer
  subgraph "Presentation Layer"
    Workbook["Tableau Workbook:\nCanadian Variable Mortgage Rate Study 2015-2024.twbx"]:::presentation
    Server["Tableau Server\n(Publishes Extracts, Dashboards)"]:::presentation
    SlideDeck["Slide Deck:\nCanadian Variable Mortgage Rate Study Presentation.pptx"]:::presentation
  end

  %% Documentation & Licensing
  subgraph "Documentation & Versioning"
    README["Project README.md"]:::external
    LICENSE["Project LICENSE"]:::external
  end

  %% Users & External Interfaces
  subgraph "Users & External Interfaces"
    Users["Stakeholders:\nFinancial & Policy Analysts"]:::external
    API["API / Export Endpoints"]:::external
  end

  %% Data Flow
  BOC -->|extract| ETL
  Housing -->|extract| ETL
  Scheduler -->|triggers daily @02:00| ETL
  ETL -->|load to staging| Staging
  SQLJobs -->|transform & load| Fact
  SQLJobs -->|transform & load| Dim
  SQLJobs -->|process staging| Staging
  Staging -->|data access| Stats
  Fact -->|data access| Stats
  Stats -->|write results| Fact
  Scheduler -->|triggers| Proc
  Proc -->|write results| Fact
  Fact -->|extracts| Server
  Server -->|publishes extracts| Workbook
  Workbook -->|feeds overview| SlideDeck
  Server -->|interactive dashboards| Users
  Server -->|API endpoints| API

  %% Click Events
  click Workbook "https://github.com/m03jam14/insightengine/blob/main/Canadian Variable Mortgage Rate Study 2015-2024.twbx"
  click SlideDeck "https://github.com/m03jam14/insightengine/blob/main/Canadian Variable Mortgage Rate Study Presentation.pptx"
  click README "https://github.com/m03jam14/insightengine/blob/main/README.md"
  click LICENSE "https://github.com/m03jam14/insightengine/tree/main/LICENSE"

  %% Styles
  classDef storage fill:#b8e986,stroke:#4d7c0f,stroke-width:2px;
  classDef compute fill:#f4b400,stroke:#a66c00,stroke-width:2px;
  classDef presentation fill:#4285f4,stroke:#0f4fa8,stroke-width:2px;
  classDef external fill:#e0e0e0,stroke:#888888,stroke-width:2px;
```
