How weiye-jing/datax-web works
1. Overall shape
This is a distributed DataX orchestration platform split into two main Spring Boot applications: datax-admin as the control plane and datax-executor as the worker plane. They share common job/RPC contracts from datax-core, and the repo also includes a lower-level datax-rpc module that implements a richer RPC/remoting stack.
2. Control plane: datax-admin
datax-admin is the web console and scheduling center. It exposes controllers for users, projects, data sources, templates, jobs, logs, registry info, and JSON generation. It also bundles a compiled frontend in src/main/resources/static, so the admin UI is served directly from the backend jar.
3. Persistence and metadata
The admin module persists the platform state in MySQL using MyBatis-Plus/MyBatis mapper XMLs. The main stored entities are job definitions, datasource definitions, job groups, logs, templates, projects, users, roles, permissions, and registry records. The SQL schema is shipped in bin/db/datax_web.sql.
4. JSON construction subsystem
A large part of the admin module is a DataX configuration builder. The tool/datax, tool/query, tool/meta, and tool/pojo packages inspect source/target databases, infer table/column metadata, and generate DataX JSON for supported systems such as MySQL, Oracle, PostgreSQL, SQL Server, Hive, HBase, MongoDB, and ClickHouse. This is the main path for “select datasource → inspect schema → generate DataX job config”.
5. Scheduling and trigger subsystem
The admin module contains an embedded scheduling engine under core/scheduler, core/trigger, core/route, and core/thread. These components implement cron-based triggering, job dispatch, executor route selection, failover, registry monitoring, log reporting, and failure monitoring. In architectural terms, this is the scheduler responsible for deciding *when* and *where* a job should run.
6. Execution plane: datax-executor
datax-executor is the remote worker process. It exposes job handlers for running DataX jobs and non-DataX script jobs, including shell, Python, and PowerShell. It also includes timeout/kill handling and log parsing/statistics helpers, so the executor is responsible for actually launching processes and reporting progress back.
7. Runtime flow from schedule to execution
The primary runtime path is: user configures a job in the admin UI → admin stores job metadata and generated JSON → scheduler triggers a job → a selected executor receives the trigger → executor runs DataX or a script process → executor streams logs/summary back → admin records logs, status, and run statistics. The admin UI then uses those persisted logs and reports to show live progress and history.
8. Registry and discovery
Executors periodically register themselves, and the admin side monitors registry state to keep an up-to-date view of online executors. This enables cluster deployment and routing strategies such as first/last/random/round-robin/consistent-hash/LFU/LRU/failover/busy-over selection, which are implemented in the admin routing package.
9. Shared job contract layer: datax-core
datax-core contains the shared interfaces and models that define the job/executor contract: AdminBiz, ExecutorBiz, trigger/request/response models, callback payloads, registry parameters, and return types. It also contains thread utilities, logging utilities, glue/script job handler abstractions, and remote-call helpers. This module is the boundary that keeps admin and executor speaking the same protocol.
10. RPC/remoting layer: datax-rpc
datax-rpc is a standalone remoting library with multiple transport and serialization options. It includes Netty, Jetty, Mina, and servlet-based transports, plus serializers such as Hessian, Jackson, Kryo, and Protostuff. Architecturally, it provides the infrastructure for remote service invocation and service registration beyond the specific admin/executor job flow.
11. Security and API access
The admin application uses JWT-based authentication/authorization filters and a security configuration. Swagger is also protected by token validation. This means operational APIs and job-management endpoints are not just open web pages; they are secured backend services.
12. Operational concerns
The platform is designed for cluster deployment and long-running batch jobs. It includes timeout enforcement, kill hooks, failure retries, failure callbacks, log cleanup, and log/report aggregation. These are important diagram elements because they show that execution is not fire-and-forget; it is continuously monitored and controlled.
13. Packaging and deployment
Maven modules are assembled into deployable distributions using datax-assembly and per-app assembly descriptors. Each app also has startup/stop shell scripts under bin/ or src/main/bin/, indicating a server-style deployment model rather than a library-only project.
14. External systems
The main external dependencies that materially shape the architecture are:
- MySQL for persistent metadata and scheduling state
- DataX as the actual data movement engine
- Source/target databases and platforms such as RDBMS, Hive, HBase, ClickHouse, MongoDB
- Optional script runtimes for shell/Python/PowerShell jobs
15. How to draw it
A good architecture diagram would show: Web UI → admin service → database + scheduler → executor cluster → DataX/script runtime → source/target data systems, with shared protocol/model modules underneath and a separate RPC/remoting foundation supporting the admin-to-executor communication path.
Main entry points: DataXAdminApplication.java Admin app · DataXExecutorApplication.java Executor app · ExecutorJobHandler.java Job handlers · JobScheduler.java Scheduler
Components 21
Control plane
Admin UI, APIs, metadata, and scheduler
- Web UI · Compiled static admin console served from the backend jar
datax-admin/src/main/resources/static/index.html - Admin app · Control plane application exposing job, user, datasource, and registry APIs
datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java - Security · JWT authentication, authorization, and Swagger access protection
datax-admin/src/main/java/com/wugui/datax/admin/config/SecurityConfig.java - Metadata store · Job, datasource, user, role, log, template, and registry persistence
datax-admin/src/main/resources/mybatis-mapper/JobInfoMapper.xml - JSON builder · Inspects schemas and generates DataX job JSON for supported sources and targets
datax-admin/src/main/java/com/wugui/datax/admin/tool/datax/DataxJsonHelper.java - Datasource tools · Database metadata, query, and POJO helpers for supported systems
datax-admin/src/main/java/com/wugui/datax/admin/tool/meta/DatabaseMetaFactory.java - Scheduler · Cron scheduling and trigger orchestration
datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java - Routing · Executor selection strategies such as round-robin, random, failover, LFU, and LRU
datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java - Registry monitor · Tracks online executors and registry heartbeats
datax-admin/src/main/java/com/wugui/datax/admin/core/thread/JobRegistryMonitorHelper.java - Admin biz · Admin-side implementation of the shared AdminBiz contract
datax-admin/src/main/java/com/wugui/datax/admin/service/impl/AdminBizImpl.java
Worker plane
Executor runtime and job handlers
- Executor app · Remote worker process for DataX and script jobs
datax-executor/src/main/java/com/wugui/datax/executor/DataXExecutorApplication.java - Job handlers · Runs DataX jobs and shell, Python, or PowerShell script jobs
datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/ExecutorJobHandler.java - Kill handler · Stops timed-out or canceled worker processes
datax-executor/src/main/java/com/wugui/datax/executor/service/jobhandler/KillJobHandler.java - Log parser · Extracts execution statistics and progress from worker logs
datax-executor/src/main/java/com/wugui/datax/executor/service/logparse/LogStatistics.java
Shared core
Common job contracts and executor helpers
- Core contracts · Shared AdminBiz, ExecutorBiz, trigger, registry, and result models
datax-core/src/main/java/com/wugui/datatx/core/biz/AdminBiz.java - Core runtime · Job threading, callbacks, logging, glue handlers, and utility helpers
datax-core/src/main/java/com/wugui/datatx/core/executor/JobExecutor.java
RPC layer
Remoting and service registry infrastructure
- Remoting stack · Transport, serialization, load balancing, and service registration infrastructure
datax-rpc/src/main/java/com/wugui/datax/rpc/remoting/invoker/XxlRpcInvokerFactory.java
External systems
Databases, DataX runtime, and script runtimes
- DataX engine · Actual batch data movement engine invoked by generated jobs
- Source/target systems · MySQL, Oracle, PostgreSQL, SQL Server, Hive, HBase, MongoDB, ClickHouse
- Script runtimes · Shell, Python, and PowerShell execution environments
Deployment
Packaging and startup assets
- Assemblies · Packaging and startup scripts for admin and executor distributions
datax-assembly/package.xml
Connections 22
- Web UI → Admin app: uses UI · Static frontend is served by the admin application
- Admin app → Security: secured by · Admin APIs are protected by JWT filters and security config
- Admin app → Metadata store: persists · Admin stores jobs, users, logs, registry state, and metadata
- Admin app → JSON builder: builds JSON · Admin exposes DataX configuration generation endpoints
- JSON builder → Datasource tools: inspects schema · JSON generation relies on metadata and query helpers
- Admin app → Scheduler: schedules · Admin drives cron-based triggering and dispatch
- Scheduler → Routing: selects executor · Scheduler routes each trigger using a strategy
- Scheduler → Registry monitor: tracks cluster · Scheduler consults registry state for online executors
- Admin app → Admin biz: implements · Admin side exposes the shared AdminBiz service
- Admin biz → Core contracts: depends on · AdminBiz is defined in the shared contract layer
- Executor app → Job handlers: runs jobs · Executor application hosts the job execution handlers
- Executor app → Kill handler: can stop · Executor supports process termination and timeout handling
- Executor app → Log parser: reports logs · Executor parses logs into progress and summary data
- Executor app → Core contracts: uses contract · Executor side also relies on the shared API models
- Job handlers → DataX engine: launches · Worker handlers invoke the DataX runtime for ETL jobs
- Job handlers → Script runtimes: launches · Worker handlers can execute non-DataX scripts
- Job handlers → Source/target systems: moves data · Executed jobs read from and write to external data systems
- Executor app → Remoting stack: remoted via · Executor communicates with admin through the RPC/remoting layer
- Admin app → Remoting stack: remoted via · Admin also uses the remoting stack for cluster communication
- Remoting stack → Core contracts: transports · RPC stack carries shared request, response, and callback models
- Assemblies → Admin app: packages · Assembly scripts package the admin service for deployment
- Assemblies → Executor app: packages · Assembly scripts package the executor service for deployment