datus-ai/datus-agent

1. Top-level shape

- This repo is a Python 3.12 application centered on the datus package.

- It is not a single app: it bundles a reusable core engine plus multiple delivery surfaces: CLI, FastAPI API, MCP server/client, and a web chatbot.

- The architecture is layered around “agent + workflow + tools + storage + interfaces”.

2. Primary execution core: the agent/workflow engine

- datus/agent contains the orchestration layer that executes configured workflows made of nodes.

- Nodes are the basic units of work and include core steps like schema linking, SQL generation, SQL execution, reasoning, reflection, output formatting, and several agentic subflows.

- Workflow definitions live in YAML (datus/agent/workflow.yml), and workflow_runner.py / plan.py coordinate execution order, parallel branches, and subworkflows.

3. Node-based boundary

- The datus/agent/node directory defines a clear boundary between orchestration and behavior.

- Nodes are specialized by function: query generation, document search, metric search, dashboard/report generation, skill creation, scheduler support, and human-in-the-loop/feedback handling.

- This makes the workflow graph extensible without changing the higher-level interface layers.

4. Knowledge and retrieval layer

- datus/storage is the persistent context layer powering the repo’s “evolvable context” concept.

- It stores multiple knowledge types: schema metadata, reference SQL, reference templates, semantic models, metrics, external knowledge, feedback, task state, and subject trees.

- LanceDB is used for vector storage, while SQLite appears as a local relational backend for supporting state and metadata.

5. Document ingestion pipeline

- datus/storage/document implements a full ingestion flow: fetch content from local files, GitHub, or web sources; resolve doc site structure; clean and parse HTML/Markdown; chunk content; and store it.

- This is the main path for turning platform docs or domain docs into retrievable context for the agent.

- The ingestion flow is separate from query execution, so documentation knowledge can evolve independently from database interactions.

6. Model/provider abstraction

- datus/models wraps many LLM providers behind a common interface: OpenAI, Claude, Gemini, DeepSeek, Qwen, Kimi, MiniMax, GLM, OpenRouter, and Codex.

- The repo supports per-node model assignment, so different workflow steps can use different models.

- There is also session management, streaming support, and LiteLLM-style compatibility layers, which matters for how requests and responses flow through the system.

7. Tools as the action layer

- datus/tools is the execution capability layer used by nodes and subagents.

- It includes database connectors, function tools, BI/dashboard assembly tools, date parsing, semantic tools, platform doc search, filesystem and permission tools, MCP tools, and skill tools.

- The tools layer is where the agent touches external systems or performs deterministic operations, while nodes decide when and how to invoke those tools.

8. Database and semantic adapter boundary

- datus/tools/db_tools and datus/storage/semantic_model / metric separate raw SQL execution from semantic-layer-driven querying.

- The repo supports built-in SQLite and DuckDB, plus pluggable external adapters for systems like PostgreSQL, MySQL, Snowflake, Trino, Spark, ClickHouse, and others.

- This boundary is important: the agent can either work directly against physical tables or translate business metrics through a semantic model.

9. User-facing interfaces

- datus/cli is the primary interactive surface, with REPL, command-based flows, TUI/web helpers, datasource setup, model selection, plan mode, skill management, and subagent creation.

- datus/api exposes the same capabilities over FastAPI through route/service layers for chat, databases, knowledge base, models, CLI operations, MCP, visualization, and success stories.

- The web chatbot under datus/cli/web is a lightweight browser UI layered on top of the same chat execution logic.

10. MCP and gateway integration

- datus/mcp_server.py, datus/tools/mcp_tools, and datus/api/routes/mcp_routes.py show that MCP is a first-class integration path.

- The repo can both expose Datus as an MCP server and consume external MCP tools from the CLI.

- datus/gateway adds another integration boundary for chat platforms such as Slack and Feishu, translating incoming messages into the internal runtime and formatting responses back out.

11. Auth, config, and environment

- Configuration is file-driven, centered on conf/agent.yml.example and loader code in datus/configuration.

- Auth is split into provider auth, OAuth/token storage, and legacy auth compatibility in datus/api/auth and datus/auth.

- Environment variables are used for secret injection, while permission/profile modules constrain which tools and actions a user or subagent can access.

12. Feedback, validation, and continuous improvement loop

- datus/validation, datus/api/services/chat_task_manager.py, and feedback-related storage/routes support evaluation and post-run learning.

- The repo explicitly closes the loop: chat/session output, user feedback, SQL fixes, skill validation, and benchmark runs all feed back into the knowledge base.

- This is a key architectural theme: the system is not only generating SQL, it is accumulating reusable context.

13. Benchmarking and offline evaluation

- benchmark/ and tests/regression/ support SQL accuracy evaluation, multi-round benchmarks, and dataset-driven comparisons such as Spider and BIRD variants.

- This subsystem is separate from runtime inference and exists to measure how workflow, prompts, models, and context changes affect output quality.

- For a diagram, it should be shown as an offline analysis path that reads from the same agent/config/model stack but does not serve end users.

14. Operational structure

- CI/CD and release automation live in .github/workflows, ci/, build_scripts/, and packaging files like pyproject.toml, Makefile, and requirements*.txt.

- The repo includes docs generation, package build scripts, release readiness checks, and extensive unit/integration/regression tests.

- The overall architecture is therefore “library + application platform”: a reusable core engine with multiple interfaces, many adapters, and strong evaluation/build infrastructure.