# amilheir/dataintegrator architecture

> Architecture diagram of the GitHub repository amilheir/dataintegrator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amilheir/dataintegrator
- Repository: https://github.com/amilheir/dataintegrator

## Overview

Data Integration is an IRIS-based, AI-assisted data integration application. A user works in the five-tab frontend to discover and profile sources, choose a deterministic loading strategy, generate or edit transformation SQL, and deploy and run a pipeline. The application also supports IntegratedML, publishing table-backed MCP tools, and embedding, searching, and chatting with a knowledge graph. The README documents these major flows; source excerpts were limited to a standalone AutoML model, so unverified internal call relationships are omitted.

Main entry points: [`src-iris/frontend/index.html`](https://github.com/amilheir/dataintegrator/blob/main/src-iris/frontend/index.html) (Five-tab UI), [`src-iris/src/DataIntegrator/Host/Orchestrator.cls`](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Host/Orchestrator.cls) (Pipeline orchestration), [`src-iris/src/DataIntegrator/KG/Ingest.cls`](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Ingest.cls) (Knowledge ingestion), [`src-iris/src/DataIntegrator/MCP/Publisher.cls`](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/MCP/Publisher.cls) (MCP publisher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_frontend["Five-tab UI<br/>[index.html]"]
  node_dispatcher["REST API<br/>[Dispatcher.cls]"]
  node_websocket["Live updates<br/>[Hub.cls]"]
end

subgraph group_integration["Data integration"]
  node_agent["Wizard agent<br/>[Architect.cls]"]
  node_discovery["Source discovery"]
  node_profiling["Source profiling<br/>[CsvProfiling.cls]"]
  node_strategy["Strategy scoring<br/>[Strategy.cls]"]
  node_sqlgen["SQL generation<br/>[SqlGen.cls]"]
  node_data_tools["Integration tools"]
end

subgraph group_runtime["Pipeline runtime"]
  node_orchestrator["Pipeline orchestration<br/>[Orchestrator.cls]"]
  node_executor["Pipeline executor<br/>[Executor.cls]"]
  node_pipeline_state[("Pipeline and run state<br/>[Pipeline.cls]")]
  node_event_log[("Replayable event log<br/>[Event.cls]")]
  node_iris_store[("IRIS data store")]
end

subgraph group_knowledge["Knowledge and AI"]
  node_knowledge_ingest["Knowledge ingestion<br/>[Ingest.cls]"]
  node_embedder["Text embedding<br/>[Embedder.cls]"]
  node_knowledge_chat["Knowledge chat<br/>[Chat.cls]"]
  node_graph_extract["Entity extraction<br/>[GraphExtract.cls]"]
end

subgraph group_extensions["Models and tools"]
  node_mcp["MCP publisher<br/>[Publisher.cls]"]
  node_automl_model["Weather regressor<br/>[inmet_temp_hgb.py]"]
  node_ollama["Ollama models"]
  node_mcp_client(("MCP clients"))
end

node_user(("User"))

node_user -->|"uses"| node_frontend
node_frontend -->|"discovers sources"| node_discovery
node_frontend -->|"profiles data"| node_profiling
node_frontend -->|"scores strategies"| node_strategy
node_frontend -->|"generates SQL"| node_sqlgen
node_frontend -->|"calls tools directly"| node_data_tools
node_frontend -->|"sends run messages"| node_agent
node_agent -->|"drives tools"| node_data_tools
node_frontend -->|"deploys pipelines"| node_orchestrator
node_orchestrator -.->|"runs pipeline"| node_executor
node_executor -.->|"loads data"| node_iris_store
node_orchestrator -.->|"manages state"| node_pipeline_state
node_frontend -->|"receives live updates"| node_websocket
node_websocket -->|"replays events"| node_event_log
node_frontend -->|"publishes tools"| node_mcp
node_mcp -->|"queries tables"| node_iris_store
node_mcp_client -->|"uses published tools"| node_mcp
node_frontend -->|"embeds text"| node_knowledge_ingest
node_knowledge_ingest -->|"creates vectors"| node_embedder
node_embedder -->|"requests embeddings"| node_ollama
node_knowledge_ingest -->|"extracts entities"| node_graph_extract
node_graph_extract -->|"requests extraction"| node_ollama
node_frontend -->|"asks questions"| node_knowledge_chat
node_knowledge_chat -->|"requests answers"| node_ollama
node_frontend -.->|"trains and predicts"| node_automl_model

click node_frontend "https://github.com/amilheir/dataintegrator/blob/main/src-iris/frontend/index.html"
click node_dispatcher "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/API/Dispatcher.cls"
click node_websocket "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/WS/Hub.cls"
click node_agent "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/Architect.cls"
click node_discovery "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/SourceDiscovery.cls"
click node_profiling "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/CsvProfiling.cls"
click node_strategy "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/Strategy.cls"
click node_sqlgen "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/SqlGen.cls"
click node_data_tools "https://github.com/amilheir/dataintegrator/tree/main/src-iris/src/DataIntegrator/Tools"
click node_orchestrator "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Host/Orchestrator.cls"
click node_executor "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Runtime/Executor.cls"
click node_pipeline_state "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Model/Pipeline.cls"
click node_event_log "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Model/Event.cls"
click node_mcp "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/MCP/Publisher.cls"
click node_knowledge_ingest "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Ingest.cls"
click node_embedder "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Embedder.cls"
click node_knowledge_chat "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Chat.cls"
click node_graph_extract "https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/GraphExtract.cls"
click node_automl_model "https://github.com/amilheir/dataintegrator/blob/main/src-iris/mlmodels/regression/inmet_temp_hgb.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_dispatcher,node_websocket,node_user toneBlue
class node_agent,node_discovery,node_profiling,node_strategy,node_sqlgen,node_data_tools toneAmber
class node_orchestrator,node_executor,node_pipeline_state,node_event_log,node_iris_store toneMint
class node_knowledge_ingest,node_embedder,node_knowledge_chat,node_graph_extract toneRose
class node_mcp,node_automl_model,node_ollama,node_mcp_client toneIndigo
```

## Components

### User interface

- [Five-tab UI](https://github.com/amilheir/dataintegrator/blob/main/src-iris/frontend/index.html): `src-iris/frontend/index.html`
- [REST API](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/API/Dispatcher.cls): `src-iris/src/DataIntegrator/API/Dispatcher.cls`
- [Live updates](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/WS/Hub.cls): `src-iris/src/DataIntegrator/WS/Hub.cls`

### Data integration

- [Wizard agent](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/Architect.cls): `src-iris/src/DataIntegrator/Agent/Architect.cls`
- [Source discovery](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/SourceDiscovery.cls): `src-iris/src/DataIntegrator/Tools/SourceDiscovery.cls`
- [Source profiling](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/CsvProfiling.cls): `src-iris/src/DataIntegrator/Tools/CsvProfiling.cls`
- [Strategy scoring](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Tools/Strategy.cls): `src-iris/src/DataIntegrator/Tools/Strategy.cls`
- [SQL generation](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/SqlGen.cls): `src-iris/src/DataIntegrator/Agent/SqlGen.cls`
- [Integration tools](https://github.com/amilheir/dataintegrator/tree/main/src-iris/src/DataIntegrator/Tools): `src-iris/src/DataIntegrator/Tools`

### Pipeline runtime

- [Pipeline orchestration](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Host/Orchestrator.cls): `src-iris/src/DataIntegrator/Host/Orchestrator.cls`
- [Pipeline executor](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Runtime/Executor.cls): `src-iris/src/DataIntegrator/Runtime/Executor.cls`
- [Pipeline and run state](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Model/Pipeline.cls): `src-iris/src/DataIntegrator/Model/Pipeline.cls`
- [Replayable event log](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Model/Event.cls): `src-iris/src/DataIntegrator/Model/Event.cls`
- **IRIS data store**

### Knowledge and AI

- [Knowledge ingestion](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Ingest.cls): `src-iris/src/DataIntegrator/KG/Ingest.cls`
- [Text embedding](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Embedder.cls): `src-iris/src/DataIntegrator/KG/Embedder.cls`
- [Knowledge chat](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/KG/Chat.cls): `src-iris/src/DataIntegrator/KG/Chat.cls`
- [Entity extraction](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/Agent/GraphExtract.cls): `src-iris/src/DataIntegrator/Agent/GraphExtract.cls`

### Models and tools

- [MCP publisher](https://github.com/amilheir/dataintegrator/blob/main/src-iris/src/DataIntegrator/MCP/Publisher.cls): `src-iris/src/DataIntegrator/MCP/Publisher.cls`
- [Weather regressor](https://github.com/amilheir/dataintegrator/blob/main/src-iris/mlmodels/regression/inmet_temp_hgb.py): `src-iris/mlmodels/regression/inmet_temp_hgb.py`
- **Ollama models**
- **MCP clients**

### Other components

- **User**

## Connections

- User → Five-tab UI: uses (evidence: `README.md`)
- Five-tab UI → Source discovery: discovers sources (evidence: `README.md`)
- Five-tab UI → Source profiling: profiles data (evidence: `README.md`)
- Five-tab UI → Strategy scoring: scores strategies (evidence: `README.md`)
- Five-tab UI → SQL generation: generates SQL (evidence: `README.md`)
- Five-tab UI → Integration tools: calls tools directly (evidence: `README.md`)
- Five-tab UI → Wizard agent: sends run messages (evidence: `README.md`)
- Wizard agent → Integration tools: drives tools (evidence: `README.md`)
- Five-tab UI → Pipeline orchestration: deploys pipelines (evidence: `README.md`)
- Pipeline orchestration → Pipeline executor: runs pipeline
- Pipeline executor → IRIS data store: loads data
- Pipeline orchestration → Pipeline and run state: manages state
- Five-tab UI → Live updates: receives live updates (evidence: `README.md`)
- Live updates → Replayable event log: replays events (evidence: `README.md`)
- Five-tab UI → MCP publisher: publishes tools (evidence: `README.md`)
- MCP publisher → IRIS data store: queries tables (evidence: `README.md`)
- MCP clients → MCP publisher: uses published tools (evidence: `README.md`)
- Five-tab UI → Knowledge ingestion: embeds text (evidence: `README.md`)
- Knowledge ingestion → Text embedding: creates vectors (evidence: `README.md`)
- Text embedding → Ollama models: requests embeddings (evidence: `README.md`)
- Knowledge ingestion → Entity extraction: extracts entities (evidence: `README.md`)
- Entity extraction → Ollama models: requests extraction (evidence: `README.md`)
- Five-tab UI → Knowledge chat: asks questions (evidence: `README.md`)
- Knowledge chat → Ollama models: requests answers (evidence: `README.md`)
- Five-tab UI → Weather regressor: trains and predicts