How key1lee/fineract-business-intelligence works
Apache Fineract Business Intelligence ingests operational data from one or more Fineract PostgreSQL sources, using either scheduled Python extraction or Kafka CDC, then transforms and serves it for office-scoped dashboards. Airflow and the control plane manage source registration, ingestion choice, CDC connector reconciliation, and migration workflows. The tree does not include warehouse or dbt model files, so those capabilities are represented at their documented directory level; edges rely on README statements where implementation excerpts are unavailable.
Main entry points: tasks.py Control Tasks · extractor.py Python Extractor · warehouse Analytics Warehouse · reconciler.py Connector Reconciler
Components 20
Ingestion
- Python Extractor
extractor/extractor.py - Watermarks
extractor/watermark_manager.py - Debezium CDC
ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java - Data Contract
controlplane/contract.py - Kafka Sink Worker
ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java - Warehouse Writer
ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java
Control Plane
- Airflow Workflows
airflow/dags/fbi_ingestion.py - Control Tasks
controlplane/tasks.py - Source Registry
controlplane/registry.py - Engine Ownership
controlplane/ownership_repo.py - Connector Reconciler
controlplane/reconciler.py - Connector Config
controlplane/connectors.py
Analytics
- Analytics Warehouse
warehouse - dbt Models
dbt
BI Serving
- Superset Dashboards
docker/superset/bootstrap_superset_assets.py - Dashboard Security
docker/superset/bootstrap_superset_security.py
Other components
- Fineract PostgreSQL
- Dashboard Users
- Kafka
- Kafka Connect
Connections 24
- Fineract PostgreSQL → Python Extractor: read rows
- Python Extractor → Analytics Warehouse: write extracts
- Python Extractor → Watermarks: track progress
extractor/extractor.py - Fineract PostgreSQL → Debezium CDC: stream changes
- Debezium CDC → Kafka: publish events
- Kafka → Kafka Sink Worker: deliver events
- Kafka Sink Worker → Warehouse Writer: dispatch writes
ingestion/ingestion-sink/src/main/java/org/apache/fineract/bi/ingestion/app/SinkWorker.java - Warehouse Writer → Analytics Warehouse: write events
- Data Contract → Python Extractor: define columns
extractor/extractor.py - Data Contract → Debezium CDC: define columns
ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/cdc/DebeziumEnvelopeParser.java - Data Contract → Warehouse Writer: define writes
ingestion/ingestion-core/src/main/java/org/apache/fineract/bi/ingestion/sink/WarehouseWriter.java - Airflow Workflows → Control Tasks: invoke tasks
airflow/dags/fbi_ingestion.py - Control Tasks → Python Extractor: run ingestion
controlplane/tasks.py - Control Tasks → Source Registry: manage sources
controlplane/tasks.py - Control Tasks → Engine Ownership: manage cutovers
controlplane/tasks.py - Control Tasks → Connector Reconciler: reconcile connectors
controlplane/tasks.py - Connector Reconciler → Source Registry: read sources
controlplane/reconciler.py - Connector Reconciler → Connector Config: build configs
controlplane/reconciler.py - Connector Reconciler → Kafka Connect: manage connectors
- Analytics Warehouse → dbt Models: transform data
- dbt Models → Analytics Warehouse: publish models
- Analytics Warehouse → Superset Dashboards: serve datasets
- Dashboard Security → Superset Dashboards: secure dashboards
- Superset Dashboards → Dashboard Users: present dashboards