# leonxu6/datasteward architecture

> Architecture diagram of the GitHub repository leonxu6/datasteward, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leonxu6/datasteward
- Repository: https://github.com/leonxu6/datasteward

## Overview

DataSteward is a self-hosted governed data platform where an agent or external MCP client asks questions through a shared governance kernel rather than receiving direct database access. The main path runs from a user through the LangGraph agent and governed tools to governed data and a response. The graph also retains the documented ingestion, warehouse/modeling, orchestration, writeback, and governance-console capabilities. Some subsystems are represented at their documented directory paths because their implementation excerpts were not provided; those links rely on README claims rather than inspected call sites.

Main entry points: [`src/dm/tools`](https://github.com/leonxu6/datasteward/tree/main/src/dm/tools) (Governed tools), [`src/dm/agent/graph.py`](https://github.com/leonxu6/datasteward/blob/main/src/dm/agent/graph.py) (LangGraph agent), [`src/dm/warehouse/store.py`](https://github.com/leonxu6/datasteward/blob/main/src/dm/warehouse/store.py) (StarRocks warehouse), [`src/dm/app/app.py`](https://github.com/leonxu6/datasteward/blob/main/src/dm/app/app.py) (Streamlit console)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User surfaces"]
  node_agent["LangGraph agent<br/>[graph.py]"]
  node_channels["DingTalk channel<br/>[dingtalk.py]"]
  node_console["Streamlit console<br/>[app.py]"]
  node_mcp["MCP server<br/>[mcp_server.py]"]
end

subgraph group_data["Data platform"]
  node_connectors["Source connectors"]
  node_u8_sync["U8 batch sync<br/>[u8_mapping.py]"]
  node_cdc["Flink CDC"]
  node_source_pg[("Postgres source")]
  node_warehouse[("StarRocks warehouse<br/>[store.py]")]
  node_modeling["dbt modeling"]
  node_dataset_catalog["Dataset lineage<br/>[model.py]"]
  node_metrics["Metrics layer<br/>[metrics.py]"]
  node_documents["Document RAG<br/>[documents.py]"]
  node_knowledge_graph[("Knowledge graph")]
end

subgraph group_kernel["Governance kernel"]
  node_tools["Governed tools"]
  node_policy["Access policy<br/>[policy.py]"]
  node_principal["Access principal<br/>[principal.py]"]
  node_audit["Append-only audit<br/>[audit.py]"]
  node_actions["Ontology actions<br/>[actions.py]"]
end

subgraph group_operations["Platform operations"]
  node_orchestration["Dagster orchestration"]
end

node_user(("User"))
node_mcp_client(("MCP client"))
node_llm(("OpenAI-compatible LLM"))

node_user -->|"asks"| node_agent
node_agent -->|"reasons with"| node_llm
node_agent -->|"calls in-process"| node_tools
node_tools -->|"returns results"| node_agent
node_agent -->|"serves through"| node_channels
node_user -->|"uses"| node_console
node_console -->|"offers agent UI"| node_agent
node_mcp_client -->|"connects via"| node_mcp
node_mcp -->|"delegates to"| node_tools
node_connectors -.->|"extracts from"| node_source_pg
node_u8_sync -->|"batch loads"| node_warehouse
node_source_pg -->|"streams changes"| node_cdc
node_cdc -->|"replicates to"| node_warehouse
node_warehouse -->|"feeds models"| node_modeling
node_modeling -->|"defines measures for"| node_metrics
node_modeling -->|"registers lineage"| node_dataset_catalog
node_metrics -->|"serves governed queries"| node_tools
node_warehouse -->|"serves governed reads"| node_tools
node_documents -->|"serves search"| node_tools
node_knowledge_graph -->|"serves graph queries"| node_tools
node_tools -->|"enforces access"| node_policy
node_tools -->|"uses identity"| node_principal
node_tools -->|"records calls"| node_audit
node_audit -->|"stores log data"| node_warehouse
node_tools -->|"exposes writebacks"| node_actions
node_actions -->|"writes after approval"| node_source_pg
node_orchestration -->|"orchestrates builds"| node_modeling
node_console -->|"surfaces lineage"| node_dataset_catalog
node_console -->|"replays governance"| node_audit

click node_agent "https://github.com/leonxu6/datasteward/blob/main/src/dm/agent/graph.py"
click node_channels "https://github.com/leonxu6/datasteward/blob/main/src/dm/channels/dingtalk.py"
click node_console "https://github.com/leonxu6/datasteward/blob/main/src/dm/app/app.py"
click node_mcp "https://github.com/leonxu6/datasteward/blob/main/src/dm/connector/mcp_server.py"
click node_connectors "https://github.com/leonxu6/datasteward/tree/main/src/dm/connect"
click node_u8_sync "https://github.com/leonxu6/datasteward/blob/main/src/dm/connect/u8_mapping.py"
click node_cdc "https://github.com/leonxu6/datasteward/tree/main/src/dm/pipeline"
click node_warehouse "https://github.com/leonxu6/datasteward/blob/main/src/dm/warehouse/store.py"
click node_modeling "https://github.com/leonxu6/datasteward/tree/main/transform/dbt"
click node_dataset_catalog "https://github.com/leonxu6/datasteward/blob/main/src/dm/datasets/model.py"
click node_metrics "https://github.com/leonxu6/datasteward/blob/main/src/dm/ontology/metrics.py"
click node_documents "https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/documents.py"
click node_knowledge_graph "https://github.com/leonxu6/datasteward/tree/main/src/dm/kg"
click node_tools "https://github.com/leonxu6/datasteward/tree/main/src/dm/tools"
click node_policy "https://github.com/leonxu6/datasteward/blob/main/src/dm/security/policy.py"
click node_principal "https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/principal.py"
click node_audit "https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/audit.py"
click node_actions "https://github.com/leonxu6/datasteward/blob/main/src/dm/ontology/actions.py"
click node_orchestration "https://github.com/leonxu6/datasteward/tree/main/src/dm/orchestration"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agent,node_channels,node_console,node_mcp,node_user,node_mcp_client toneBlue
class node_connectors,node_u8_sync,node_cdc,node_source_pg,node_warehouse,node_modeling,node_dataset_catalog,node_metrics,node_documents,node_knowledge_graph toneAmber
class node_tools,node_policy,node_principal,node_audit,node_actions toneMint
class node_orchestration toneRose
class node_llm toneIndigo
```

## Components

### User surfaces

- [LangGraph agent](https://github.com/leonxu6/datasteward/blob/main/src/dm/agent/graph.py): `src/dm/agent/graph.py`
- [DingTalk channel](https://github.com/leonxu6/datasteward/blob/main/src/dm/channels/dingtalk.py): `src/dm/channels/dingtalk.py`
- [Streamlit console](https://github.com/leonxu6/datasteward/blob/main/src/dm/app/app.py): `src/dm/app/app.py`
- [MCP server](https://github.com/leonxu6/datasteward/blob/main/src/dm/connector/mcp_server.py): `src/dm/connector/mcp_server.py`

### Data platform

- [Source connectors](https://github.com/leonxu6/datasteward/tree/main/src/dm/connect): `src/dm/connect`
- [U8 batch sync](https://github.com/leonxu6/datasteward/blob/main/src/dm/connect/u8_mapping.py): `src/dm/connect/u8_mapping.py`
- [Flink CDC](https://github.com/leonxu6/datasteward/tree/main/src/dm/pipeline): `src/dm/pipeline`
- **Postgres source**
- [StarRocks warehouse](https://github.com/leonxu6/datasteward/blob/main/src/dm/warehouse/store.py): `src/dm/warehouse/store.py`
- [dbt modeling](https://github.com/leonxu6/datasteward/tree/main/transform/dbt): `transform/dbt`
- [Dataset lineage](https://github.com/leonxu6/datasteward/blob/main/src/dm/datasets/model.py): `src/dm/datasets/model.py`
- [Metrics layer](https://github.com/leonxu6/datasteward/blob/main/src/dm/ontology/metrics.py): `src/dm/ontology/metrics.py`
- [Document RAG](https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/documents.py): `src/dm/tools/documents.py`
- [Knowledge graph](https://github.com/leonxu6/datasteward/tree/main/src/dm/kg): `src/dm/kg`

### Governance kernel

- [Governed tools](https://github.com/leonxu6/datasteward/tree/main/src/dm/tools): `src/dm/tools`
- [Access policy](https://github.com/leonxu6/datasteward/blob/main/src/dm/security/policy.py): `src/dm/security/policy.py`
- [Access principal](https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/principal.py): `src/dm/tools/principal.py`
- [Append-only audit](https://github.com/leonxu6/datasteward/blob/main/src/dm/tools/audit.py): `src/dm/tools/audit.py`
- [Ontology actions](https://github.com/leonxu6/datasteward/blob/main/src/dm/ontology/actions.py): `src/dm/ontology/actions.py`

### Platform operations

- [Dagster orchestration](https://github.com/leonxu6/datasteward/tree/main/src/dm/orchestration): `src/dm/orchestration`

### Other components

- **User**
- **MCP client**
- **OpenAI-compatible LLM**

## Connections

- User → LangGraph agent: asks (evidence: `README.md`)
- LangGraph agent → OpenAI-compatible LLM: reasons with (evidence: `README.md`)
- LangGraph agent → Governed tools: calls in-process (evidence: `src/dm/agent/graph.py`)
- Governed tools → LangGraph agent: returns results (evidence: `README.md`)
- LangGraph agent → DingTalk channel: serves through (evidence: `README.md`)
- User → Streamlit console: uses (evidence: `README.md`)
- Streamlit console → LangGraph agent: offers agent UI
- MCP client → MCP server: connects via (evidence: `README.md`)
- MCP server → Governed tools: delegates to (evidence: `src/dm/connector/mcp_server.py`)
- Source connectors → Postgres source: extracts from (evidence: `README.md`)
- U8 batch sync → StarRocks warehouse: batch loads (evidence: `src/dm/connect/u8_mapping.py`)
- Postgres source → Flink CDC: streams changes (evidence: `README.md`)
- Flink CDC → StarRocks warehouse: replicates to (evidence: `README.md`)
- StarRocks warehouse → dbt modeling: feeds models (evidence: `README.md`)
- dbt modeling → Metrics layer: defines measures for (evidence: `README.md`)
- dbt modeling → Dataset lineage: registers lineage (evidence: `src/dm/datasets/model.py`)
- Metrics layer → Governed tools: serves governed queries (evidence: `README.md`)
- StarRocks warehouse → Governed tools: serves governed reads (evidence: `README.md`)
- Document RAG → Governed tools: serves search (evidence: `README.md`)
- Knowledge graph → Governed tools: serves graph queries (evidence: `README.md`)
- Governed tools → Access policy: enforces access (evidence: `src/dm/tools/data.py`)
- Governed tools → Access principal: uses identity (evidence: `src/dm/tools/__init__.py`)
- Governed tools → Append-only audit: records calls (evidence: `src/dm/tools/data.py`)
- Append-only audit → StarRocks warehouse: stores log data (evidence: `src/dm/tools/audit.py`)
- Governed tools → Ontology actions: exposes writebacks (evidence: `src/dm/tools/__init__.py`)
- Ontology actions → Postgres source: writes after approval (evidence: `README.md`)
- Dagster orchestration → dbt modeling: orchestrates builds (evidence: `README.md`)
- Streamlit console → Dataset lineage: surfaces lineage (evidence: `README.md`)
- Streamlit console → Append-only audit: replays governance (evidence: `README.md`)