# weiye-jing/datax-web architecture

> Architecture diagram of the GitHub repository weiye-jing/datax-web, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weiye-jing/datax-web
- Repository: https://github.com/weiye-jing/datax-web

## Overview

1. **Overall shape**
   This is a distributed DataX orchestration platform split into two main Spring Boot applications: **datax-admin** as the control plane and **datax-executor** as the worker plane. They share common job/RPC contracts from **datax-core**, and the repo also includes a lower-level **datax-rpc** module that implements a richer RPC/remoting stack.

2. **Control plane: `datax-admin`**
   `datax-admin` is the web console and scheduling center. It exposes controllers for users, projects, data sources, templates, jobs, logs, registry info, and JSON generation. It also bundles a compiled frontend in `src/main/resources/static`, so the admin UI is served directly from the backend jar.

3. **Persistence and metadata**
   The admin module persists the platform state in MySQL using MyBatis-Plus/MyBatis mapper XMLs. The main stored entities are job definitions, datasource definitions, job groups, logs, templates, projects, users, roles, permissions, and registry records. The SQL schema is shipped in `bin/db/datax_web.sql`.

4. **JSON construction subsystem**
   A large part of the admin module is a DataX configuration builder. The `tool/datax`, `tool/query`, `tool/meta`, and `tool/pojo` packages inspect source/target databases, infer table/column metadata, and generate DataX JSON for supported systems such as MySQL, Oracle, PostgreSQL, SQL Server, Hive, HBase, MongoDB, and ClickHouse. This is the main path for “select datasource → inspect schema → generate DataX job config”.

5. **Scheduling and trigger subsystem**
   The admin module contains an embedded scheduling engine under `core/scheduler`, `core/trigger`, `core/route`, and `core/thread`. These components implement cron-based triggering, job dispatch, executor route selection, failover, registry monitoring, log reporting, and failure monitoring. In architectural terms, this is the scheduler responsible for deciding *when* and *where* a job should run.

6. **Execution plane: `datax-executor`**
   `datax-executor` is the remote worker process. It exposes job handlers for running DataX jobs and non-DataX script jobs, including shell, Python, and PowerShell. It also includes timeout/kill handling and log parsing/statistics helpers, so the executor is responsible for actually launching processes and reporting progress back.

7. **Runtime flow from schedule to execution**
   The primary runtime path is: user configures a job in the admin UI → admin stores job metadata and generated JSON → scheduler triggers a job → a selected executor receives the trigger → executor runs DataX or a script process → executor streams logs/summary back → admin records logs, status, and run statistics. The admin UI then uses those persisted logs and reports to show live progress and history.

8. **Registry and discovery**
   Executors periodically register themselves, and the admin side monitors registry state to keep an up-to-date view of online executors. This enables cluster deployment and routing strategies such as first/last/random/round-robin/consistent-hash/LFU/LRU/failover/busy-over selection, which are implemented in the admin routing package.

9. **Shared job contract layer: `datax-core`**
   `datax-core` contains the shared interfaces and models that define the job/executor contract: `AdminBiz`, `ExecutorBiz`, trigger/request/response models, callback payloads, registry parameters, and return types. It also contains thread utilities, logging utilities, glue/script job handler abstractions, and remote-call helpers. This module is the boundary that keeps admin and executor speaking the same protocol.

10. **RPC/remoting layer: `datax-rpc`**
    `datax-rpc` is a standalone remoting library with multiple transport and serialization options. It includes Netty, Jetty, Mina, and servlet-based transports, plus serializers such as Hessian, Jackson, Kryo, and Protostuff. Architecturally, it provides the infrastructure for remote service invocation and service registration beyond the specific admin/executor job flow.

11. **Security and API access**
    The admin application uses JWT-based authentication/authorization filters and a security configuration. Swagger is also protected by token validation. This means operational APIs and job-management endpoints are not just open web pages; they are secured backend services.

12. **Operational concerns**
    The platform is designed for cluster deployment and long-running batch jobs. It includes timeout enforcement, kill hooks, failure retries, failure callbacks, log cleanup, and log/report aggregation. These are important diagram elements because they show that execution is not fire-and-forget; it is continuously monitored and controlled.

13. **Packaging and deployment**
    Maven modules are assembled into deployable distributions using `datax-assembly` and per-app assembly descriptors. Each app also has startup/stop shell scripts under `bin/` or `src/main/bin/`, indicating a server-style deployment model rather than a library-only project.

14. **External systems**
    The main external dependencies that materially shape the architecture are:
    - **MySQL** for persistent metadata and scheduling state
    - **DataX** as the actual data movement engine
    - Source/target databases and platforms such as **RDBMS, Hive, HBase, ClickHouse, MongoDB**
    - Optional script runtimes for shell/Python/PowerShell jobs

15. **How to draw it**
    A good architecture diagram would show: **Web UI → admin service → database + scheduler → executor cluster → DataX/script runtime → source/target data systems**, with shared protocol/model modules underneath and a separate RPC/remoting foundation supporting the admin-to-executor communication path.

Main entry points: [`datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java`](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java) (Admin app), [`datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java`](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java) (Executor app), [`datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java`](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java) (Job handlers), [`datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java`](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java) (Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Control plane"]
  node_n1["Web UI<br/>frontend<br/>[index.html]"]
  node_n2["Admin app<br/>spring-boot"]
  node_n3["Security<br/>jwt-auth"]
  node_n4[("Metadata store<br/>mysql-mybatis<br/>[JobInfoMapper.xml]")]
  node_n5["JSON builder<br/>datax-config"]
  node_n6["Datasource tools<br/>schema-inspect"]
  node_n7(("Scheduler<br/>trigger-engine<br/>[JobScheduler.java]"))
  node_n8{{"Routing"}}
  node_n9(("Registry monitor<br/>cluster-state"))
  node_n10["Admin biz<br/>rpc-service<br/>[AdminBizImpl.java]"]
end

subgraph group_g2["Worker plane"]
  node_n11["Executor app<br/>spring-boot"]
  node_n12["Job handlers<br/>job-runtime"]
  node_n13["Kill handler<br/>lifecycle-control"]
  node_n14["Log parser<br/>run-summary<br/>[LogStatistics.java]"]
end

subgraph group_g3["Shared core"]
  node_n15["Core contracts<br/>shared-api<br/>[AdminBiz.java]"]
  node_n16["Core runtime<br/>executor-support<br/>[JobExecutor.java]"]
end

subgraph group_g4["RPC layer"]
  node_n17["Remoting stack<br/>rpc-library"]
end

subgraph group_g5["External systems"]
  node_n18{{"DataX engine<br/>external-runtime"}}
  node_n19[("Source/target systems<br/>datastores")]
  node_n20["Script runtimes<br/>processes"]
end

subgraph group_g6["Deployment"]
  node_n21["Assemblies<br/>deployment<br/>[package.xml]"]
end

node_n1 -->|"uses UI"| node_n2
node_n2 -->|"secured by"| node_n3
node_n2 -->|"persists"| node_n4
node_n2 -->|"builds JSON"| node_n5
node_n5 -->|"inspects schema"| node_n6
node_n2 -->|"schedules"| node_n7
node_n7 -->|"selects executor"| node_n8
node_n7 -->|"tracks cluster"| node_n9
node_n2 -->|"implements"| node_n10
node_n10 -->|"depends on"| node_n15
node_n11 -->|"runs jobs"| node_n12
node_n11 -->|"can stop"| node_n13
node_n11 -->|"reports logs"| node_n14
node_n11 -->|"uses contract"| node_n15
node_n12 -->|"launches"| node_n18
node_n12 -->|"launches"| node_n20
node_n12 -->|"moves data"| node_n19
node_n11 -->|"remoted via"| node_n17
node_n2 -->|"remoted via"| node_n17
node_n17 -->|"transports"| node_n15
node_n21 -->|"packages"| node_n2
node_n21 -->|"packages"| node_n11

click node_n1 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/resources/static/index.html"
click node_n2 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java"
click node_n3 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/config/SecurityConfig.java"
click node_n4 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/resources/mybatis-mapper/JobInfoMapper.xml"
click node_n5 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/tool/datax/DataxJsonHelper.java"
click node_n6 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/tool/meta/DatabaseMetaFactory.java"
click node_n7 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java"
click node_n8 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java"
click node_n9 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/thread/JobRegistryMonitorHelper.java"
click node_n10 "https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/service/impl/AdminBizImpl.java"
click node_n11 "https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java"
click node_n12 "https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java"
click node_n13 "https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/KillJobHandler.java"
click node_n14 "https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/logparse/LogStatistics.java"
click node_n15 "https://github.com/weiye-jing/datax-web/blob/master/datax-core/src/main/java/com/wugui/datatx/core/biz/AdminBiz.java"
click node_n16 "https://github.com/weiye-jing/datax-web/blob/master/datax-core/src/main/java/com/wugui/datatx/core/executor/JobExecutor.java"
click node_n17 "https://github.com/weiye-jing/datax-web/blob/master/datax-rpc/src/main/java/com/wugui/datax/rpc/remoting/invoker/XxlRpcInvokerFactory.java"
click node_n21 "https://github.com/weiye-jing/datax-web/blob/master/datax-assembly/package.xml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5,node_n6,node_n7,node_n8,node_n9,node_n10 toneBlue
class node_n11,node_n12,node_n13,node_n14 toneAmber
class node_n15,node_n16 toneMint
class node_n17 toneRose
class node_n18,node_n19,node_n20 toneIndigo
class node_n21 toneTeal
```

## Components

### Control plane

Admin UI, APIs, metadata, and scheduler

- [Web UI](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/resources/static/index.html): frontend · Compiled static admin console served from the backend jar · `datax-admin/src/main/resources/static/index.html`
- [Admin app](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java): spring-boot · Control plane application exposing job, user, datasource, and registry APIs · `datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java`
- [Security](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/config/SecurityConfig.java): jwt-auth · JWT authentication, authorization, and Swagger access protection · `datax-admin/src/main/java/com/wugui/datax/admin/config/SecurityConfig.java`
- [Metadata store](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/resources/mybatis-mapper/JobInfoMapper.xml): mysql-mybatis · Job, datasource, user, role, log, template, and registry persistence · `datax-admin/src/main/resources/mybatis-mapper/JobInfoMapper.xml`
- [JSON builder](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/tool/datax/DataxJsonHelper.java): datax-config · Inspects schemas and generates DataX job JSON for supported sources and targets · `datax-admin/src/main/java/com/wugui/datax/admin/tool/datax/DataxJsonHelper.java`
- [Datasource tools](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/tool/meta/DatabaseMetaFactory.java): schema-inspect · Database metadata, query, and POJO helpers for supported systems · `datax-admin/src/main/java/com/wugui/datax/admin/tool/meta/DatabaseMetaFactory.java`
- [Scheduler](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java): trigger-engine · Cron scheduling and trigger orchestration · `datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java`
- [Routing](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java): executor-routing · Executor selection strategies such as round-robin, random, failover, LFU, and LRU · `datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java`
- [Registry monitor](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/core/thread/JobRegistryMonitorHelper.java): cluster-state · Tracks online executors and registry heartbeats · `datax-admin/src/main/java/com/wugui/datax/admin/core/thread/JobRegistryMonitorHelper.java`
- [Admin biz](https://github.com/weiye-jing/datax-web/blob/master/datax-admin/src/main/java/com/wugui/datax/admin/service/impl/AdminBizImpl.java): rpc-service · Admin-side implementation of the shared AdminBiz contract · `datax-admin/src/main/java/com/wugui/datax/admin/service/impl/AdminBizImpl.java`

### Worker plane

Executor runtime and job handlers

- [Executor app](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java): spring-boot · Remote worker process for DataX and script jobs · `datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java`
- [Job handlers](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java): job-runtime · Runs DataX jobs and shell, Python, or PowerShell script jobs · `datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java`
- [Kill handler](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/KillJobHandler.java): lifecycle-control · Stops timed-out or canceled worker processes · `datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/KillJobHandler.java`
- [Log parser](https://github.com/weiye-jing/datax-web/blob/master/datax-executor/src/main/java/com/wugui/datax/executor/service/logparse/LogStatistics.java): run-summary · Extracts execution statistics and progress from worker logs · `datax-executor/src/main/java/com/wugui/datax/executor/service/logparse/LogStatistics.java`

### Shared core

Common job contracts and executor helpers

- [Core contracts](https://github.com/weiye-jing/datax-web/blob/master/datax-core/src/main/java/com/wugui/datatx/core/biz/AdminBiz.java): shared-api · Shared AdminBiz, ExecutorBiz, trigger, registry, and result models · `datax-core/src/main/java/com/wugui/datatx/core/biz/AdminBiz.java`
- [Core runtime](https://github.com/weiye-jing/datax-web/blob/master/datax-core/src/main/java/com/wugui/datatx/core/executor/JobExecutor.java): executor-support · Job threading, callbacks, logging, glue handlers, and utility helpers · `datax-core/src/main/java/com/wugui/datatx/core/executor/JobExecutor.java`

### RPC layer

Remoting and service registry infrastructure

- [Remoting stack](https://github.com/weiye-jing/datax-web/blob/master/datax-rpc/src/main/java/com/wugui/datax/rpc/remoting/invoker/XxlRpcInvokerFactory.java): rpc-library · Transport, serialization, load balancing, and service registration infrastructure · `datax-rpc/src/main/java/com/wugui/datax/rpc/remoting/invoker/XxlRpcInvokerFactory.java`

### External systems

Databases, DataX runtime, and script runtimes

- **DataX engine**: external-runtime · Actual batch data movement engine invoked by generated jobs
- **Source/target systems**: datastores · MySQL, Oracle, PostgreSQL, SQL Server, Hive, HBase, MongoDB, ClickHouse
- **Script runtimes**: processes · Shell, Python, and PowerShell execution environments

### Deployment

Packaging and startup assets

- [Assemblies](https://github.com/weiye-jing/datax-web/blob/master/datax-assembly/package.xml): deployment · Packaging and startup scripts for admin and executor distributions · `datax-assembly/package.xml`

## Connections

- Web UI → Admin app: uses UI · Static frontend is served by the admin application
- Admin app → Security: secured by · Admin APIs are protected by JWT filters and security config
- Admin app → Metadata store: persists · Admin stores jobs, users, logs, registry state, and metadata
- Admin app → JSON builder: builds JSON · Admin exposes DataX configuration generation endpoints
- JSON builder → Datasource tools: inspects schema · JSON generation relies on metadata and query helpers
- Admin app → Scheduler: schedules · Admin drives cron-based triggering and dispatch
- Scheduler → Routing: selects executor · Scheduler routes each trigger using a strategy
- Scheduler → Registry monitor: tracks cluster · Scheduler consults registry state for online executors
- Admin app → Admin biz: implements · Admin side exposes the shared AdminBiz service
- Admin biz → Core contracts: depends on · AdminBiz is defined in the shared contract layer
- Executor app → Job handlers: runs jobs · Executor application hosts the job execution handlers
- Executor app → Kill handler: can stop · Executor supports process termination and timeout handling
- Executor app → Log parser: reports logs · Executor parses logs into progress and summary data
- Executor app → Core contracts: uses contract · Executor side also relies on the shared API models
- Job handlers → DataX engine: launches · Worker handlers invoke the DataX runtime for ETL jobs
- Job handlers → Script runtimes: launches · Worker handlers can execute non-DataX scripts
- Job handlers → Source/target systems: moves data · Executed jobs read from and write to external data systems
- Executor app → Remoting stack: remoted via · Executor communicates with admin through the RPC/remoting layer
- Admin app → Remoting stack: remoted via · Admin also uses the remoting stack for cluster communication
- Remoting stack → Core contracts: transports · RPC stack carries shared request, response, and callback models
- Assemblies → Admin app: packages · Assembly scripts package the admin service for deployment
- Assemblies → Executor app: packages · Assembly scripts package the executor service for deployment
