# yuyuejia/datay architecture

> Architecture diagram of the GitHub repository yuyuejia/datay, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuyuejia/datay
- Repository: https://github.com/yuyuejia/datay

## Overview

DataY is a lightweight data platform: users configure sources and ETL flows, execute and schedule jobs, model data and metrics, then query dashboards or ask questions in natural language. The map also preserves the documented data-map, CDC, warehouse, and MCP capabilities. Edges use README statements for documented product flows and sampled source excerpts for visible service calls; unsampled implementations are represented only at their supported directory paths.

Main entry points: [`datay-core/src/main/java/com/data/job/ETLFlowTask.java`](https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/job/ETLFlowTask.java) (Flow Runtime), [`datay-web/src/main/java/com/data/datafusion/ai/assistant`](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/ai/assistant) (AI Assistants), [`datay-web/src/main/java/com/data/datafusion/mcp`](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/mcp) (MCP Endpoint), [`datay-core/src/main/java/com/data/job/component`](https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/job/component) (ETL Designer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_integration["Data Integration"]
  node_etl["ETL Designer"]
  node_sources["Source Connectors"]
  node_metadata["Schema Metadata"]
  node_cdc["CDC Collectors"]
end

subgraph group_execution["Execution and Storage"]
  node_core["Flow Runtime<br/>[ETLFlowTask.java]"]
  node_scheduler["Job Scheduler"]
  node_status[("Run Status")]
  node_file["File Storage"]
end

subgraph group_warehouse["Modeling and Analytics"]
  node_warehouse["Analytical Engines<br/>[DuckDBEngine.java]"]
  node_model[("Dimensional Models")]
  node_metric["Metric Queries"]
  node_dashboard["Datasets and Dashboards"]
  node_datamap["Data Map"]
end

subgraph group_intelligence["AI and MCP"]
  node_assistant["AI Assistants"]
  node_rag["Metric RAG"]
  node_tools["Agent Tools"]
  node_mcp["MCP Endpoint"]
end

subgraph group_platform["Platform Services"]
  node_query["Source Query Service"]
  node_llm["LLM Configuration"]
end

node_user(("Platform User"))
node_externaldb[("External Data Stores")]
node_llmservice["LLM Provider"]
node_agentclient(("MCP Client"))

node_user -->|"designs flows"| node_etl
node_etl -->|"configures"| node_core
node_core -->|"runs connectors"| node_sources
node_core -->|"runs CDC"| node_cdc
node_sources -.->|"reads schemas"| node_metadata
node_core -->|"processes data"| node_warehouse
node_scheduler -->|"dispatches jobs"| node_core
node_core -.->|"records status"| node_status
node_core -->|"writes results"| node_file
node_warehouse -->|"loads data"| node_externaldb
node_user -->|"defines models"| node_model
node_model -->|"supplies semantics"| node_metric
node_user -->|"builds dashboards"| node_dashboard
node_dashboard -->|"executes SQL"| node_query
node_query -->|"queries"| node_externaldb
node_user -->|"queries metrics"| node_metric
node_user -->|"searches assets"| node_datamap
node_user -->|"asks questions"| node_assistant
node_assistant -->|"retrieves context"| node_rag
node_assistant -.->|"uses tools"| node_tools
node_assistant -->|"queries metrics"| node_metric
node_llm -.->|"configures client"| node_llmservice
node_agentclient -->|"invokes tools"| node_mcp
node_mcp -->|"shares query logic"| node_assistant
node_mcp -->|"manages tasks"| node_etl
node_mcp -->|"manages models"| node_model

click node_etl "https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/job/component"
click node_sources "https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/datasource/provider"
click node_metadata "https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/metadata"
click node_cdc "https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/job/component/cdc"
click node_core "https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/job/ETLFlowTask.java"
click node_scheduler "https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/service/scheduler"
click node_status "https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/status/StatusStorageStrategy.java"
click node_file "https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/file/FileStorageStrategy.java"
click node_warehouse "https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/job/DuckDBEngine.java"
click node_model "https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/domain"
click node_metric "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/MetricQueryService.java"
click node_dashboard "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/dashboard/DatasetQueryService.java"
click node_datamap "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/datamap/DataMapQueryService.java"
click node_assistant "https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/ai/assistant"
click node_rag "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/ai/rag/MetricRagService.java"
click node_tools "https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/ai/tool"
click node_mcp "https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/mcp"
click node_query "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/DataSourceQueryService.java"
click node_llm "https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/ai/llm/LlmConfigService.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_etl,node_sources,node_metadata,node_cdc,node_user,node_agentclient toneBlue
class node_core,node_scheduler,node_status,node_file,node_externaldb toneAmber
class node_warehouse,node_model,node_metric,node_dashboard,node_datamap toneMint
class node_assistant,node_rag,node_tools,node_mcp toneRose
class node_query,node_llm,node_llmservice toneIndigo
```

## Components

### Data Integration

- [ETL Designer](https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/job/component): `datay-core/src/main/java/com/data/job/component`
- [Source Connectors](https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/datasource/provider): `datay-core/src/main/java/com/data/datasource/provider`
- [Schema Metadata](https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/metadata): `datay-core/src/main/java/com/data/metadata`
- [CDC Collectors](https://github.com/yuyuejia/datay/tree/main/datay-core/src/main/java/com/data/job/component/cdc): `datay-core/src/main/java/com/data/job/component/cdc`

### Execution and Storage

- [Flow Runtime](https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/job/ETLFlowTask.java): `datay-core/src/main/java/com/data/job/ETLFlowTask.java`
- [Job Scheduler](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/service/scheduler): `datay-web/src/main/java/com/data/datafusion/service/scheduler`
- [Run Status](https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/status/StatusStorageStrategy.java): `datay-core/src/main/java/com/data/status/StatusStorageStrategy.java`
- [File Storage](https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/file/FileStorageStrategy.java): `datay-core/src/main/java/com/data/file/FileStorageStrategy.java`

### Modeling and Analytics

- [Analytical Engines](https://github.com/yuyuejia/datay/blob/main/datay-core/src/main/java/com/data/job/DuckDBEngine.java): `datay-core/src/main/java/com/data/job/DuckDBEngine.java`
- [Dimensional Models](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/domain): `datay-web/src/main/java/com/data/datafusion/domain`
- [Metric Queries](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/MetricQueryService.java): `datay-web/src/main/java/com/data/datafusion/service/MetricQueryService.java`
- [Datasets and Dashboards](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/dashboard/DatasetQueryService.java): `datay-web/src/main/java/com/data/datafusion/service/dashboard/DatasetQueryService.java`
- [Data Map](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/datamap/DataMapQueryService.java): `datay-web/src/main/java/com/data/datafusion/service/datamap/DataMapQueryService.java`

### AI and MCP

- [AI Assistants](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/ai/assistant): `datay-web/src/main/java/com/data/datafusion/ai/assistant`
- [Metric RAG](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/ai/rag/MetricRagService.java): `datay-web/src/main/java/com/data/datafusion/ai/rag/MetricRagService.java`
- [Agent Tools](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/ai/tool): `datay-web/src/main/java/com/data/datafusion/ai/tool`
- [MCP Endpoint](https://github.com/yuyuejia/datay/tree/main/datay-web/src/main/java/com/data/datafusion/mcp): `datay-web/src/main/java/com/data/datafusion/mcp`

### Platform Services

- [Source Query Service](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/service/DataSourceQueryService.java): `datay-web/src/main/java/com/data/datafusion/service/DataSourceQueryService.java`
- [LLM Configuration](https://github.com/yuyuejia/datay/blob/main/datay-web/src/main/java/com/data/datafusion/ai/llm/LlmConfigService.java): `datay-web/src/main/java/com/data/datafusion/ai/llm/LlmConfigService.java`

### Other components

- **Platform User**
- **External Data Stores**
- **LLM Provider**
- **MCP Client**

## Connections

- Platform User → ETL Designer: designs flows
- ETL Designer → Flow Runtime: configures
- Flow Runtime → Source Connectors: runs connectors
- Flow Runtime → CDC Collectors: runs CDC
- Source Connectors → Schema Metadata: reads schemas (evidence: `datay-core/src/main/java/com/data/datasource/provider/TdengineDataSourceProvider.java`)
- Flow Runtime → Analytical Engines: processes data
- Job Scheduler → Flow Runtime: dispatches jobs
- Flow Runtime → Run Status: records status
- Flow Runtime → File Storage: writes results
- Analytical Engines → External Data Stores: loads data
- Platform User → Dimensional Models: defines models
- Dimensional Models → Metric Queries: supplies semantics (evidence: `datay-web/src/main/java/com/data/datafusion/service/MetricQueryService.java`)
- Platform User → Datasets and Dashboards: builds dashboards
- Datasets and Dashboards → Source Query Service: executes SQL (evidence: `datay-web/src/main/java/com/data/datafusion/service/dashboard/DatasetQueryService.java`)
- Source Query Service → External Data Stores: queries (evidence: `datay-web/src/main/java/com/data/datafusion/service/DataSourceQueryService.java`)
- Platform User → Metric Queries: queries metrics
- Platform User → Data Map: searches assets
- Platform User → AI Assistants: asks questions
- AI Assistants → Metric RAG: retrieves context (evidence: `datay-web/src/main/java/com/data/datafusion/ai/rag/MetricRagService.java`)
- AI Assistants → Agent Tools: uses tools
- AI Assistants → Metric Queries: queries metrics
- LLM Configuration → LLM Provider: configures client
- MCP Client → MCP Endpoint: invokes tools
- MCP Endpoint → AI Assistants: shares query logic
- MCP Endpoint → ETL Designer: manages tasks
- MCP Endpoint → Dimensional Models: manages models
