How amilheir/dataintegrator works
Data Integration is an IRIS-based, AI-assisted data integration application. A user works in the five-tab frontend to discover and profile sources, choose a deterministic loading strategy, generate or edit transformation SQL, and deploy and run a pipeline. The application also supports IntegratedML, publishing table-backed MCP tools, and embedding, searching, and chatting with a knowledge graph. The README documents these major flows; source excerpts were limited to a standalone AutoML model, so unverified internal call relationships are omitted.
Main entry points: index.html Five-tab UI · Orchestrator.cls Pipeline orchestration · Ingest.cls Knowledge ingestion · Publisher.cls MCP publisher
Components 23
User interface
- Five-tab UI
src-iris/frontend/index.html - REST API
src-iris/src/DataIntegrator/API/Dispatcher.cls - Live updates
src-iris/src/DataIntegrator/WS/Hub.cls
Data integration
- Wizard agent
src-iris/src/DataIntegrator/Agent/Architect.cls - Source discovery
src-iris/src/DataIntegrator/Tools/SourceDiscovery.cls - Source profiling
src-iris/src/DataIntegrator/Tools/CsvProfiling.cls - Strategy scoring
src-iris/src/DataIntegrator/Tools/Strategy.cls - SQL generation
src-iris/src/DataIntegrator/Agent/SqlGen.cls - Integration tools
src-iris/src/DataIntegrator/Tools
Pipeline runtime
- Pipeline orchestration
src-iris/src/DataIntegrator/Host/Orchestrator.cls - Pipeline executor
src-iris/src/DataIntegrator/Runtime/Executor.cls - Pipeline and run state
src-iris/src/DataIntegrator/Model/Pipeline.cls - Replayable event log
src-iris/src/DataIntegrator/Model/Event.cls - IRIS data store
Knowledge and AI
- Knowledge ingestion
src-iris/src/DataIntegrator/KG/Ingest.cls - Text embedding
src-iris/src/DataIntegrator/KG/Embedder.cls - Knowledge chat
src-iris/src/DataIntegrator/KG/Chat.cls - Entity extraction
src-iris/src/DataIntegrator/Agent/GraphExtract.cls
Models and tools
- MCP publisher
src-iris/src/DataIntegrator/MCP/Publisher.cls - Weather regressor
src-iris/mlmodels/regression/inmet_temp_hgb.py - Ollama models
- MCP clients
Other components
- User
Connections 25
- User → Five-tab UI: uses
README.md - Five-tab UI → Source discovery: discovers sources
README.md - Five-tab UI → Source profiling: profiles data
README.md - Five-tab UI → Strategy scoring: scores strategies
README.md - Five-tab UI → SQL generation: generates SQL
README.md - Five-tab UI → Integration tools: calls tools directly
README.md - Five-tab UI → Wizard agent: sends run messages
README.md - Wizard agent → Integration tools: drives tools
README.md - Five-tab UI → Pipeline orchestration: deploys pipelines
README.md - Pipeline orchestration → Pipeline executor: runs pipeline
- Pipeline executor → IRIS data store: loads data
- Pipeline orchestration → Pipeline and run state: manages state
- Five-tab UI → Live updates: receives live updates
README.md - Live updates → Replayable event log: replays events
README.md - Five-tab UI → MCP publisher: publishes tools
README.md - MCP publisher → IRIS data store: queries tables
README.md - MCP clients → MCP publisher: uses published tools
README.md - Five-tab UI → Knowledge ingestion: embeds text
README.md - Knowledge ingestion → Text embedding: creates vectors
README.md - Text embedding → Ollama models: requests embeddings
README.md - Knowledge ingestion → Entity extraction: extracts entities
README.md - Entity extraction → Ollama models: requests extraction
README.md - Five-tab UI → Knowledge chat: asks questions
README.md - Knowledge chat → Ollama models: requests answers
README.md - Five-tab UI → Weather regressor: trains and predicts