# key1lee/fineract-business-intelligence architecture

> Architecture diagram of the GitHub repository key1lee/fineract-business-intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/key1lee/fineract-business-intelligence
- Repository: https://github.com/key1lee/fineract-business-intelligence

## Overview

Apache Fineract Business Intelligence ingests operational data from one or more Fineract PostgreSQL sources, using either scheduled Python extraction or Kafka CDC, then transforms and serves it for office-scoped dashboards. Airflow and the control plane manage source registration, ingestion choice, CDC connector reconciliation, and migration workflows. The tree does not include warehouse or dbt model files, so those capabilities are represented at their documented directory level; edges rely on README statements where implementation excerpts are unavailable.

Main entry points: [`controlplane/tasks.py`](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/tasks.py) (Control Tasks), [`extractor/extractor.py`](https://github.com/key1lee/fineract-business-intelligence/blob/main/extractor/extractor.py) (Python Extractor), [`warehouse`](https://github.com/key1lee/fineract-business-intelligence/tree/main/warehouse) (Analytics Warehouse), [`controlplane/reconciler.py`](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/reconciler.py) (Connector Reconciler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_extractor["Python Extractor<br/>[extractor.py]"]
  node_watermarks[("Watermarks")]
  node_cdc["Debezium CDC"]
  node_contract["Data Contract<br/>[contract.py]"]
  node_sinkworker["Kafka Sink Worker<br/>[SinkWorker.java]"]
  node_writer["Warehouse Writer"]
end

subgraph group_governance["Control Plane"]
  node_airflow["Airflow Workflows<br/>[fbi_ingestion.py]"]
  node_tasks["Control Tasks<br/>[tasks.py]"]
  node_registry[("Source Registry<br/>[registry.py]")]
  node_ownership[("Engine Ownership<br/>[ownership_repo.py]")]
  node_reconciler["Connector Reconciler<br/>[reconciler.py]"]
  node_connectors["Connector Config<br/>[connectors.py]"]
end

subgraph group_analytics["Analytics"]
  node_warehouse[("Analytics Warehouse")]
  node_dbt["dbt Models"]
end

subgraph group_bi["BI Serving"]
  node_superset["Superset Dashboards"]
  node_security["Dashboard Security"]
end

node_fineract[("Fineract PostgreSQL")]
node_users(("Dashboard Users"))
node_kafka["Kafka"]
node_connect["Kafka Connect"]

node_fineract -->|"read rows"| node_extractor
node_extractor -->|"write extracts"| node_warehouse
node_extractor -->|"track progress"| node_watermarks
node_fineract -->|"stream changes"| node_cdc
node_cdc -->|"publish events"| node_kafka
node_kafka -->|"deliver events"| node_sinkworker
node_sinkworker -->|"dispatch writes"| node_writer
node_writer -->|"write events"| node_warehouse
node_contract -->|"define columns"| node_extractor
node_contract -->|"define columns"| node_cdc
node_contract -->|"define writes"| node_writer
node_airflow -->|"invoke tasks"| node_tasks
node_tasks -->|"run ingestion"| node_extractor
node_tasks -->|"manage sources"| node_registry
node_tasks -->|"manage cutovers"| node_ownership
node_tasks -->|"reconcile connectors"| node_reconciler
node_reconciler -->|"read sources"| node_registry
node_reconciler -->|"build configs"| node_connectors
node_reconciler -.->|"manage connectors"| node_connect
node_warehouse -->|"transform data"| node_dbt
node_dbt -->|"publish models"| node_warehouse
node_warehouse -->|"serve datasets"| node_superset
node_security -->|"secure dashboards"| node_superset
node_superset -->|"present dashboards"| node_users

click node_extractor "https://github.com/key1lee/fineract-business-intelligence/blob/main/extractor/extractor.py"
click node_watermarks "https://github.com/key1lee/fineract-business-intelligence/blob/main/extractor/watermark_manager.py"
click node_cdc "https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java"
click node_contract "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/contract.py"
click node_sinkworker "https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java"
click node_writer "https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java"
click node_airflow "https://github.com/key1lee/fineract-business-intelligence/blob/main/airflow/dags/fbi_ingestion.py"
click node_tasks "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/tasks.py"
click node_registry "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/registry.py"
click node_ownership "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/ownership_repo.py"
click node_reconciler "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/reconciler.py"
click node_connectors "https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/connectors.py"
click node_warehouse "https://github.com/key1lee/fineract-business-intelligence/tree/main/warehouse"
click node_dbt "https://github.com/key1lee/fineract-business-intelligence/tree/main/dbt"
click node_superset "https://github.com/key1lee/fineract-business-intelligence/blob/main/docker/superset/bootstrap_superset_assets.py"
click node_security "https://github.com/key1lee/fineract-business-intelligence/blob/main/docker/superset/bootstrap_superset_security.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extractor,node_watermarks,node_cdc,node_contract,node_sinkworker,node_writer,node_users toneBlue
class node_airflow,node_tasks,node_registry,node_ownership,node_reconciler,node_connectors,node_fineract toneAmber
class node_warehouse,node_dbt toneMint
class node_superset,node_security toneRose
class node_kafka,node_connect toneIndigo
```

## Components

### Ingestion

- [Python Extractor](https://github.com/key1lee/fineract-business-intelligence/blob/main/extractor/extractor.py): `extractor/extractor.py`
- [Watermarks](https://github.com/key1lee/fineract-business-intelligence/blob/main/extractor/watermark_manager.py): `extractor/watermark_manager.py`
- [Debezium CDC](https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java): `ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java`
- [Data Contract](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/contract.py): `controlplane/contract.py`
- [Kafka Sink Worker](https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java): `ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java`
- [Warehouse Writer](https://github.com/key1lee/fineract-business-intelligence/blob/main/ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java): `ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java`

### Control Plane

- [Airflow Workflows](https://github.com/key1lee/fineract-business-intelligence/blob/main/airflow/dags/fbi_ingestion.py): `airflow/dags/fbi_ingestion.py`
- [Control Tasks](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/tasks.py): `controlplane/tasks.py`
- [Source Registry](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/registry.py): `controlplane/registry.py`
- [Engine Ownership](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/ownership_repo.py): `controlplane/ownership_repo.py`
- [Connector Reconciler](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/reconciler.py): `controlplane/reconciler.py`
- [Connector Config](https://github.com/key1lee/fineract-business-intelligence/blob/main/controlplane/connectors.py): `controlplane/connectors.py`

### Analytics

- [Analytics Warehouse](https://github.com/key1lee/fineract-business-intelligence/tree/main/warehouse): `warehouse`
- [dbt Models](https://github.com/key1lee/fineract-business-intelligence/tree/main/dbt): `dbt`

### BI Serving

- [Superset Dashboards](https://github.com/key1lee/fineract-business-intelligence/blob/main/docker/superset/bootstrap_superset_assets.py): `docker/superset/bootstrap_superset_assets.py`
- [Dashboard Security](https://github.com/key1lee/fineract-business-intelligence/blob/main/docker/superset/bootstrap_superset_security.py): `docker/superset/bootstrap_superset_security.py`

### Other components

- **Fineract PostgreSQL**
- **Dashboard Users**
- **Kafka**
- **Kafka Connect**

## Connections

- Fineract PostgreSQL → Python Extractor: read rows
- Python Extractor → Analytics Warehouse: write extracts
- Python Extractor → Watermarks: track progress (evidence: `extractor/extractor.py`)
- Fineract PostgreSQL → Debezium CDC: stream changes
- Debezium CDC → Kafka: publish events
- Kafka → Kafka Sink Worker: deliver events
- Kafka Sink Worker → Warehouse Writer: dispatch writes (evidence: `ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java`)
- Warehouse Writer → Analytics Warehouse: write events
- Data Contract → Python Extractor: define columns (evidence: `extractor/extractor.py`)
- Data Contract → Debezium CDC: define columns (evidence: `ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java`)
- Data Contract → Warehouse Writer: define writes (evidence: `ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java`)
- Airflow Workflows → Control Tasks: invoke tasks (evidence: `airflow/dags/fbi_ingestion.py`)
- Control Tasks → Python Extractor: run ingestion (evidence: `controlplane/tasks.py`)
- Control Tasks → Source Registry: manage sources (evidence: `controlplane/tasks.py`)
- Control Tasks → Engine Ownership: manage cutovers (evidence: `controlplane/tasks.py`)
- Control Tasks → Connector Reconciler: reconcile connectors (evidence: `controlplane/tasks.py`)
- Connector Reconciler → Source Registry: read sources (evidence: `controlplane/reconciler.py`)
- Connector Reconciler → Connector Config: build configs (evidence: `controlplane/reconciler.py`)
- Connector Reconciler → Kafka Connect: manage connectors
- Analytics Warehouse → dbt Models: transform data
- dbt Models → Analytics Warehouse: publish models
- Analytics Warehouse → Superset Dashboards: serve datasets
- Dashboard Security → Superset Dashboards: secure dashboards
- Superset Dashboards → Dashboard Users: present dashboards
