# datus-ai/datus-agent architecture

> Architecture diagram of the GitHub repository datus-ai/datus-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datus-ai/datus-agent
- Repository: https://github.com/datus-ai/datus-agent

## Overview

1. **Top-level shape**
   - This repo is a Python 3.12 application centered on the `datus` package.
   - It is not a single app: it bundles a reusable core engine plus multiple delivery surfaces: CLI, FastAPI API, MCP server/client, and a web chatbot.
   - The architecture is layered around “agent + workflow + tools + storage + interfaces”.

2. **Primary execution core: the agent/workflow engine**
   - `datus/agent` contains the orchestration layer that executes configured workflows made of nodes.
   - Nodes are the basic units of work and include core steps like schema linking, SQL generation, SQL execution, reasoning, reflection, output formatting, and several agentic subflows.
   - Workflow definitions live in YAML (`datus/agent/workflow.yml`), and `workflow_runner.py` / `plan.py` coordinate execution order, parallel branches, and subworkflows.

3. **Node-based boundary**
   - The `datus/agent/node` directory defines a clear boundary between orchestration and behavior.
   - Nodes are specialized by function: query generation, document search, metric search, dashboard/report generation, skill creation, scheduler support, and human-in-the-loop/feedback handling.
   - This makes the workflow graph extensible without changing the higher-level interface layers.

4. **Knowledge and retrieval layer**
   - `datus/storage` is the persistent context layer powering the repo’s “evolvable context” concept.
   - It stores multiple knowledge types: schema metadata, reference SQL, reference templates, semantic models, metrics, external knowledge, feedback, task state, and subject trees.
   - LanceDB is used for vector storage, while SQLite appears as a local relational backend for supporting state and metadata.

5. **Document ingestion pipeline**
   - `datus/storage/document` implements a full ingestion flow: fetch content from local files, GitHub, or web sources; resolve doc site structure; clean and parse HTML/Markdown; chunk content; and store it.
   - This is the main path for turning platform docs or domain docs into retrievable context for the agent.
   - The ingestion flow is separate from query execution, so documentation knowledge can evolve independently from database interactions.

6. **Model/provider abstraction**
   - `datus/models` wraps many LLM providers behind a common interface: OpenAI, Claude, Gemini, DeepSeek, Qwen, Kimi, MiniMax, GLM, OpenRouter, and Codex.
   - The repo supports per-node model assignment, so different workflow steps can use different models.
   - There is also session management, streaming support, and LiteLLM-style compatibility layers, which matters for how requests and responses flow through the system.

7. **Tools as the action layer**
   - `datus/tools` is the execution capability layer used by nodes and subagents.
   - It includes database connectors, function tools, BI/dashboard assembly tools, date parsing, semantic tools, platform doc search, filesystem and permission tools, MCP tools, and skill tools.
   - The tools layer is where the agent touches external systems or performs deterministic operations, while nodes decide when and how to invoke those tools.

8. **Database and semantic adapter boundary**
   - `datus/tools/db_tools` and `datus/storage/semantic_model` / `metric` separate raw SQL execution from semantic-layer-driven querying.
   - The repo supports built-in SQLite and DuckDB, plus pluggable external adapters for systems like PostgreSQL, MySQL, Snowflake, Trino, Spark, ClickHouse, and others.
   - This boundary is important: the agent can either work directly against physical tables or translate business metrics through a semantic model.

9. **User-facing interfaces**
   - `datus/cli` is the primary interactive surface, with REPL, command-based flows, TUI/web helpers, datasource setup, model selection, plan mode, skill management, and subagent creation.
   - `datus/api` exposes the same capabilities over FastAPI through route/service layers for chat, databases, knowledge base, models, CLI operations, MCP, visualization, and success stories.
   - The web chatbot under `datus/cli/web` is a lightweight browser UI layered on top of the same chat execution logic.

10. **MCP and gateway integration**
   - `datus/mcp_server.py`, `datus/tools/mcp_tools`, and `datus/api/routes/mcp_routes.py` show that MCP is a first-class integration path.
   - The repo can both expose Datus as an MCP server and consume external MCP tools from the CLI.
   - `datus/gateway` adds another integration boundary for chat platforms such as Slack and Feishu, translating incoming messages into the internal runtime and formatting responses back out.

11. **Auth, config, and environment**
   - Configuration is file-driven, centered on `conf/agent.yml.example` and loader code in `datus/configuration`.
   - Auth is split into provider auth, OAuth/token storage, and legacy auth compatibility in `datus/api/auth` and `datus/auth`.
   - Environment variables are used for secret injection, while permission/profile modules constrain which tools and actions a user or subagent can access.

12. **Feedback, validation, and continuous improvement loop**
   - `datus/validation`, `datus/api/services/chat_task_manager.py`, and feedback-related storage/routes support evaluation and post-run learning.
   - The repo explicitly closes the loop: chat/session output, user feedback, SQL fixes, skill validation, and benchmark runs all feed back into the knowledge base.
   - This is a key architectural theme: the system is not only generating SQL, it is accumulating reusable context.

13. **Benchmarking and offline evaluation**
   - `benchmark/` and `tests/regression/` support SQL accuracy evaluation, multi-round benchmarks, and dataset-driven comparisons such as Spider and BIRD variants.
   - This subsystem is separate from runtime inference and exists to measure how workflow, prompts, models, and context changes affect output quality.
   - For a diagram, it should be shown as an offline analysis path that reads from the same agent/config/model stack but does not serve end users.

14. **Operational structure**
   - CI/CD and release automation live in `.github/workflows`, `ci/`, `build_scripts/`, and packaging files like `pyproject.toml`, `Makefile`, and `requirements*.txt`.
   - The repo includes docs generation, package build scripts, release readiness checks, and extensive unit/integration/regression tests.
   - The overall architecture is therefore “library + application platform”: a reusable core engine with multiple interfaces, many adapters, and strong evaluation/build infrastructure.

Main entry points: [`datus/agent/agent.py`](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/agent.py) (Agent core), [`datus/tools`](https://github.com/datus-ai/datus-agent/tree/main/datus/tools) (Tool layer), [`datus/storage`](https://github.com/datus-ai/datus-agent/tree/main/datus/storage) (Storage core), [`datus/main.py`](https://github.com/datus-ai/datus-agent/blob/main/datus/main.py) (Main entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core runtime"]
  node_agent_runtime["Agent core<br/>orchestrator<br/>[agent.py]"]
  node_workflow_runner["Workflow runner<br/>executor<br/>[workflow_runner.py]"]
  node_plan_module["Planning<br/>planner<br/>[plan.py]"]
  node_workflow_def["Workflow spec<br/>yaml config<br/>[workflow.yml]"]
  node_node_factory["Node factory<br/>node registry<br/>[node_factory.py]"]
  node_agentic_nodes{{"Agentic nodes<br/>specialized nodes"}}
  node_sql_nodes{{"SQL nodes<br/>query nodes"}}
end

subgraph group_knowledge["Knowledge layer"]
  node_storage_core[("Storage core<br/>context store")]
  node_doc_ingest["Doc ingest<br/>pipeline"]
  node_semantic_store[("Semantic data<br/>semantic store")]
  node_vector_store[("Vector store<br/>vector backend<br/>[lance_backend.py]")]
end

subgraph group_interfaces["Interfaces"]
  node_datus_main["Main entry<br/>launcher<br/>[main.py]"]
  node_cli_surface["CLI<br/>interface"]
  node_api_surface["API"]
  node_web_chatbot["Web chat<br/>browser UI"]
  node_mcp_surface["MCP<br/>protocol integration<br/>[mcp_server.py]"]
  node_gateway_surface["Gateway<br/>chat bridge"]
end

subgraph group_capabilities["Capabilities"]
  node_db_tools["DB tools<br/>connectors"]
  node_semantic_tools["Semantic tools<br/>query adapter"]
  node_model_layer["Model layer<br/>llm providers"]
  node_tool_registry["Tool layer<br/>tooling"]
  node_permission_layer["Permissions<br/>access control"]
  node_validation_loop["Validation<br/>evaluation"]
end

subgraph group_ops["Ops & eval"]
  node_benchmarking["Benchmarking<br/>offline eval"]
end

node_datus_main -->|"starts"| node_cli_surface
node_datus_main -->|"serves"| node_api_surface
node_datus_main -->|"exposes"| node_mcp_surface
node_datus_main -->|"bridges"| node_gateway_surface
node_cli_surface -->|"invokes"| node_agent_runtime
node_api_surface -->|"invokes"| node_agent_runtime
node_web_chatbot -->|"reuses"| node_agent_runtime
node_gateway_surface -->|"forwards"| node_agent_runtime
node_mcp_surface -->|"serves tools"| node_tool_registry
node_agent_runtime -->|"plans"| node_plan_module
node_plan_module -->|"reads"| node_workflow_def
node_agent_runtime -->|"executes"| node_workflow_runner
node_workflow_runner -->|"instantiates"| node_node_factory
node_node_factory -->|"builds"| node_agentic_nodes
node_node_factory -->|"builds"| node_sql_nodes
node_sql_nodes -->|"queries"| node_db_tools
node_agentic_nodes -->|"uses"| node_tool_registry
node_tool_registry -->|"includes"| node_semantic_tools
node_tool_registry -->|"constrained by"| node_permission_layer
node_tool_registry -->|"calls"| node_model_layer
node_storage_core -->|"indexes"| node_vector_store
node_storage_core -->|"stores"| node_semantic_store
node_doc_ingest -->|"feeds"| node_storage_core
node_doc_ingest -->|"embeds"| node_vector_store
node_validation_loop -->|"writes back"| node_storage_core
node_api_surface -->|"reports"| node_validation_loop
node_benchmarking -->|"measures"| node_agent_runtime
node_benchmarking -->|"compares"| node_model_layer
node_benchmarking -->|"reads"| node_storage_core

click node_datus_main "https://github.com/datus-ai/datus-agent/blob/main/datus/main.py"
click node_agent_runtime "https://github.com/datus-ai/datus-agent/blob/main/datus/agent/agent.py"
click node_workflow_runner "https://github.com/datus-ai/datus-agent/blob/main/datus/agent/workflow_runner.py"
click node_plan_module "https://github.com/datus-ai/datus-agent/blob/main/datus/agent/plan.py"
click node_workflow_def "https://github.com/datus-ai/datus-agent/blob/main/datus/agent/workflow.yml"
click node_node_factory "https://github.com/datus-ai/datus-agent/blob/main/datus/agent/node/node_factory.py"
click node_agentic_nodes "https://github.com/datus-ai/datus-agent/tree/main/datus/agent/node"
click node_sql_nodes "https://github.com/datus-ai/datus-agent/tree/main/datus/agent/node"
click node_storage_core "https://github.com/datus-ai/datus-agent/tree/main/datus/storage"
click node_doc_ingest "https://github.com/datus-ai/datus-agent/tree/main/datus/storage/document"
click node_semantic_store "https://github.com/datus-ai/datus-agent/tree/main/datus/storage/semantic_model"
click node_vector_store "https://github.com/datus-ai/datus-agent/blob/main/datus/storage/vector/lance_backend.py"
click node_db_tools "https://github.com/datus-ai/datus-agent/tree/main/datus/tools/db_tools"
click node_semantic_tools "https://github.com/datus-ai/datus-agent/tree/main/datus/tools/semantic_tools"
click node_model_layer "https://github.com/datus-ai/datus-agent/tree/main/datus/models"
click node_tool_registry "https://github.com/datus-ai/datus-agent/tree/main/datus/tools"
click node_permission_layer "https://github.com/datus-ai/datus-agent/tree/main/datus/tools/permission"
click node_validation_loop "https://github.com/datus-ai/datus-agent/tree/main/datus/validation"
click node_cli_surface "https://github.com/datus-ai/datus-agent/tree/main/datus/cli"
click node_api_surface "https://github.com/datus-ai/datus-agent/tree/main/datus/api"
click node_web_chatbot "https://github.com/datus-ai/datus-agent/tree/main/datus/cli/web"
click node_mcp_surface "https://github.com/datus-ai/datus-agent/blob/main/datus/mcp_server.py"
click node_gateway_surface "https://github.com/datus-ai/datus-agent/tree/main/datus/gateway"
click node_benchmarking "https://github.com/datus-ai/datus-agent/tree/main/benchmark"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agent_runtime,node_workflow_runner,node_plan_module,node_workflow_def,node_node_factory,node_agentic_nodes,node_sql_nodes toneBlue
class node_storage_core,node_doc_ingest,node_semantic_store,node_vector_store toneAmber
class node_datus_main,node_cli_surface,node_api_surface,node_web_chatbot,node_mcp_surface,node_gateway_surface toneMint
class node_db_tools,node_semantic_tools,node_model_layer,node_tool_registry,node_permission_layer,node_validation_loop toneRose
class node_benchmarking toneIndigo
```

## Components

### Core runtime

Agent orchestration, workflow planning, and node execution.

- [Agent core](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/agent.py): orchestrator · Coordinates configured workflows and runtime state for agentic execution. · `datus/agent/agent.py`
- [Workflow runner](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/workflow_runner.py): executor · Runs workflow graphs, handles ordering, branching, and subworkflows. · `datus/agent/workflow_runner.py`
- [Planning](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/plan.py): planner · Builds execution plans and control flow for agent runs. · `datus/agent/plan.py`
- [Workflow spec](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/workflow.yml): yaml config · Workflow definition file that wires node sequences and execution structure. · `datus/agent/workflow.yml`
- [Node factory](https://github.com/datus-ai/datus-agent/blob/main/datus/agent/node/node_factory.py): node registry · Creates concrete node implementations from workflow/node configuration. · `datus/agent/node/node_factory.py`
- [Agentic nodes](https://github.com/datus-ai/datus-agent/tree/main/datus/agent/node): specialized nodes · Specialized subflows for chat, explore, compare, feedback, dashboard, report, metrics, semantic model, skill, and scheduler tasks. · `datus/agent/node`
- [SQL nodes](https://github.com/datus-ai/datus-agent/tree/main/datus/agent/node): query nodes · Nodes for schema linking, SQL generation, reasoning, execution, reflection, fixing, and output formatting. · `datus/agent/node`

### Knowledge layer

Persistent context, retrieval, and document ingestion.

- [Storage core](https://github.com/datus-ai/datus-agent/tree/main/datus/storage): context store · Persistent context foundation for schema, metrics, templates, feedback, tasks, and subject trees. · `datus/storage`
- [Doc ingest](https://github.com/datus-ai/datus-agent/tree/main/datus/storage/document): pipeline · Fetches, cleans, parses, chunks, and stores documents from local, GitHub, or web sources. · `datus/storage/document`
- [Semantic data](https://github.com/datus-ai/datus-agent/tree/main/datus/storage/semantic_model): semantic store · Stores semantic models and metric definitions for business-layer querying. · `datus/storage/semantic_model`
- [Vector store](https://github.com/datus-ai/datus-agent/blob/main/datus/storage/vector/lance_backend.py): vector backend · LanceDB-backed embedding index used for retrieval across stored knowledge. · `datus/storage/vector/lance_backend.py`

### Interfaces

User-facing entrypoints and integration surfaces.

- [Main entry](https://github.com/datus-ai/datus-agent/blob/main/datus/main.py): launcher · Top-level application entry that dispatches into CLI, API, MCP, or gateway surfaces. · `datus/main.py`
- [CLI](https://github.com/datus-ai/datus-agent/tree/main/datus/cli): interface · Primary interactive command-line surface with REPL, setup, plan mode, model and skill management. · `datus/cli`
- [API](https://github.com/datus-ai/datus-agent/tree/main/datus/api): fastapi · FastAPI service exposing chat, KB, database, model, tool, visualization, and MCP routes. · `datus/api`
- [Web chat](https://github.com/datus-ai/datus-agent/tree/main/datus/cli/web): browser UI · Lightweight browser chatbot layered on the shared chat execution path. · `datus/cli/web`
- [MCP](https://github.com/datus-ai/datus-agent/blob/main/datus/mcp_server.py): protocol integration · MCP server/client path for exposing Datus tools and consuming external MCP tools. · `datus/mcp_server.py`
- [Gateway](https://github.com/datus-ai/datus-agent/tree/main/datus/gateway): chat bridge · Slack and Feishu bridge that translates platform messages into internal chat runtime. · `datus/gateway`

### Capabilities

Model, tool, permission, and validation subsystems.

- [DB tools](https://github.com/datus-ai/datus-agent/tree/main/datus/tools/db_tools): connectors · Physical database connectors and SQL execution helpers for SQLite, DuckDB, and external adapters. · `datus/tools/db_tools`
- [Semantic tools](https://github.com/datus-ai/datus-agent/tree/main/datus/tools/semantic_tools): query adapter · Bridges agent/tool calls to semantic-layer storage and metric-aware retrieval. · `datus/tools/semantic_tools`
- [Model layer](https://github.com/datus-ai/datus-agent/tree/main/datus/models): llm providers · Common interface over provider-specific LLM backends, session handling, and streaming. · `datus/models`
- [Tool layer](https://github.com/datus-ai/datus-agent/tree/main/datus/tools): tooling · Registry and functional tools used by nodes to search, generate, assemble, and act on external systems. · `datus/tools`
- [Permissions](https://github.com/datus-ai/datus-agent/tree/main/datus/tools/permission): access control · Profiles, permission hooks, and tool constraints that limit actions by user or subagent. · `datus/tools/permission`
- [Validation](https://github.com/datus-ai/datus-agent/tree/main/datus/validation): evaluation · Run-time checks, feedback handling, and learning loop for improving future outputs. · `datus/validation`

### Ops & eval

Benchmarking, docs, packaging, and release automation.

- [Benchmarking](https://github.com/datus-ai/datus-agent/tree/main/benchmark): offline eval · Offline benchmarking and regression analysis for SQL quality, recall, and multi-round runs. · `benchmark`

## Connections

- Main entry → CLI: starts · Launches interactive CLI flows and subcommands.
- Main entry → API: serves · Starts the HTTP API surface.
- Main entry → MCP: exposes · Runs the MCP server entrypoint.
- Main entry → Gateway: bridges · Starts external chat platform integration.
- CLI → Agent core: invokes · CLI commands trigger agent runs and workflows.
- API → Agent core: invokes · API chat and service routes dispatch into the agent runtime.
- Web chat → Agent core: reuses · Web chat shares the same execution engine as CLI/API chat.
- Gateway → Agent core: forwards · Incoming platform messages are routed into the shared runtime.
- MCP → Tool layer: serves tools · MCP exposes internal capabilities as protocol tools.
- Agent core → Planning: plans · Computes execution plans for a request.
- Planning → Workflow spec: reads · Plan logic is driven by workflow configuration.
- Agent core → Workflow runner: executes · Delegates workflow execution to the runner.
- Workflow runner → Node factory: instantiates · Creates node implementations from workflow definitions.
- Node factory → Agentic nodes: builds · Produces specialized agentic subflow nodes.
- Node factory → SQL nodes: builds · Produces core SQL and reasoning nodes.
- SQL nodes → DB tools: queries · Schema linking, SQL generation, and execution rely on database connectors.
- Agentic nodes → Tool layer: uses · Agentic nodes invoke tools for search, generation, and external actions.
- Tool layer → Semantic tools: includes · Tool layer includes semantic-aware query helpers.
- Tool layer → Permissions: constrained by · Tool availability and action scope are filtered by permissions.
- Tool layer → Model layer: calls · LLM-backed tools and prompts rely on shared model providers.
- Storage core → Vector store: indexes · Persistent knowledge is backed by vector retrieval.
- Storage core → Semantic data: stores · Semantic models and metrics are persisted as part of the knowledge base.
- Doc ingest → Storage core: feeds · Ingested docs are normalized into persistent context stores.
- Doc ingest → Vector store: embeds · Chunked content is embedded for retrieval.
- Validation → Storage core: writes back · Feedback, fixes, and validation results update stored context.
- API → Validation: reports · Chat task and feedback endpoints connect to evaluation and improvement flows.
- Benchmarking → Agent core: measures · Benchmarks exercise the same core workflows and node stack.
- Benchmarking → Model layer: compares · Offline runs vary models and prompts to assess quality.
- Benchmarking → Storage core: reads · Evaluation datasets and reference context are sourced from storage.