# alibaba/datax architecture

> Architecture diagram of the GitHub repository alibaba/datax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba/datax
- Repository: https://github.com/alibaba/datax

## Overview

1. **What this repository is**
   - This is the open-source **DataX** data integration platform: a **Java/Maven multi-module ETL framework** plus a large catalog of reader/writer plugins.
   - The repo is organized around a **core execution engine** and many **connector modules** for databases, warehouses, filesystems, graph stores, time-series systems, and Alibaba Cloud services.

2. **Top-level architecture**
   - The system has three main layers:
     - **Core runtime**: parses jobs, schedules work, runs tasks, tracks metrics.
     - **Plugin SPI**: standardized Reader/Writer interfaces and shared record/exception/config classes.
     - **Connector implementations**: one module per source/target system.
   - Architecture diagrams should show DataX as a **job orchestration engine** that sits between a source system and a sink system, with optional transformation steps in the middle.

3. **Core execution engine**
   - The `core` module contains the orchestration path:
     - `Engine` starts a job.
     - `ConfigParser` and related utilities load/validate job definitions.
     - `JobContainer` coordinates job lifecycle.
     - `TaskGroupContainer` runs parallel task groups and `TaskMonitor` supervises execution.
   - Scheduling is built around standalone/process-inner modes, with communication and reporting components under `core/statistics`.

4. **Data movement model**
   - Data flows through an internal transport pipeline:
     - Readers produce `Record` objects.
     - Records travel through an in-memory `Channel` implementation.
     - Writers consume records from the channel.
   - The transport layer is in `core/transport`, and the default channel is `MemoryChannel`.
   - This means the architecture is **push/pull between task runners**, not a direct source-to-sink call path.

5. **Plugin contract and shared runtime types**
   - The `common` module defines the plugin API surface:
     - `Reader` / `Writer` SPI interfaces.
     - `AbstractJobPlugin` / `AbstractTaskPlugin` bases.
     - `Record`, `Column` types, and casting utilities.
     - Central exceptions, error codes, configuration helpers, and plugin collectors.
   - `common` is the main boundary between the core engine and connector modules.

6. **Transformation subsystem**
   - DataX supports record-level transformations through the `transformer` module and the transformer implementations in `core/transport/transformer`.
   - Built-in transformers include filtering, substring, replace, pad, digest, and Groovy-based transformation.
   - In the architecture, transformers sit **between reader output and writer input**, inside the transport/exchanger pipeline.

7. **Connector/plugin modules**
   - Most directories at the repo root are standalone plugin modules, each packaging one source or sink integration.
   - There are broad families:
     - **RDBMS**: MySQL, Oracle, PostgreSQL, SQL Server, DRDS, Kingbase, OceanBase, GaussDB, etc.
     - **Warehouses/lake/storage**: ODPS/MaxCompute, HDFS/Hive, OSS, FTP, ClickHouse, Doris, StarRocks, Databend, SelectDB, Hologres.
     - **NoSQL/graph/time-series**: MongoDB, Cassandra, HBase variants, OTS, GDB, Neo4j, TDengine, TSDB, OpenTSDB.
     - **Alibaba Cloud middleware**: DataHub, LogHub, OCS, ADS, ADB, SLS.
   - Each plugin module usually contains:
     - a main Reader/Writer class,
     - a `Key` or `Constant` config holder,
     - error codes,
     - helper/utility classes for splitting, mapping, format conversion, or remote client access,
     - `plugin.json` and a job template.

8. **Shared connector utilities**
   - Some subsystems are factored out and reused by multiple plugins:
     - `plugin-rdbms-util` for common JDBC split, metadata, and DB helper logic.
     - `plugin-unstructured-storage-util` for file/text/binary/parquet handling and split logic.
   - These modules reduce duplication across many readers/writers and should be drawn as **shared libraries feeding several connector modules**.

9. **Plugin packaging and deployment**
   - Each plugin module has its own `pom.xml` and `src/main/assembly/package.xml`, indicating it is built and packaged separately.
   - Runtime plugin metadata is carried in `plugin.json` and `plugin_job_template.json`.
   - The `core/src/main/bin/datax.py` entrypoint and `core/src/main/conf` files indicate the distributed runtime is assembled as a runnable DataX installation with plugins dropped into a plugin directory structure.

10. **Examples, tests, and operational artifacts**
   - `datax-example` contains sample jobs and example code showing how the framework is invoked.
   - Many plugin modules include unit/integration tests under `src/test`.
   - `rpm/` contains packaging scripts for system-level distribution.
   - These are not part of the runtime data path, but they document and validate the architecture.

11. **Important architectural boundary**
   - The key boundary in this repo is between:
     - **framework concerns**: parsing, scheduling, concurrency, transport, metrics, error handling;
     - **connector concerns**: protocol specifics, schema mapping, batching, partitioning, type conversion, and remote service clients.
   - An architecture diagram should keep the core engine centralized and show each plugin module as a pluggable endpoint using the common SPI.

12. **How to read the whole repo**
   - Think of the repository as a **plugin ecosystem around a single batch data pipeline**:
     - job definition enters the core,
     - core loads matching reader/writer plugins,
     - reader tasks fetch and split source data,
     - records flow through memory and optional transformers,
     - writer tasks persist to the target system,
     - statistics/reporting track the run.

Main entry points: [`common/src/main/java/com/alibaba/datax/common`](https://github.com/alibaba/datax/tree/master/common/src/main/java/com/alibaba/datax/common) (Common SPI), [`core/src/main/bin/datax.py`](https://github.com/alibaba/datax/blob/master/core/src/main/bin/datax.py) (Packaging), [`core/src/main/java/com/alibaba/datax/core/transport`](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/transport) (Transport), [`core/src/main/java/com/alibaba/datax/core/Engine.java`](https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/Engine.java) (Engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core runtime"]
  node_engine(("Engine<br/>job entrypoint<br/>[Engine.java]"))
  node_config_parser["Config parser<br/>job config<br/>[ConfigParser.java]"]
  node_job_container["Job container<br/>lifecycle coordinator<br/>[JobContainer.java]"]
  node_scheduler["Scheduler"]
  node_task_group["Task group<br/>parallel runner"]
  node_transport["Transport<br/>record pipeline"]
  node_memory_channel["Memory channel<br/>[MemoryChannel.java]"]
  node_transformers{{"Transformers<br/>record transform"}}
  node_statistics["Statistics<br/>metrics/reporting"]
end

subgraph group_spi["Plugin SPI"]
  node_spi_common["Common SPI<br/>plugin API"]
end

subgraph group_shared["Shared utilities"]
  node_rdbms_util["RDBMS util<br/>shared jdbc lib"]
  node_storage_util["Storage util<br/>shared file lib"]
end

subgraph group_connectors["Connectors"]
  node_mysql_rdbms[("MySQL/PG<br/>relational connectors<br/>[MysqlReader.java]")]
  node_clickhouse[("ClickHouse<br/>warehouse connector")]
  node_hdfs_oss[("HDFS/OSS<br/>file connectors<br/>[HdfsReader.java]")]
  node_mongo[("MongoDB<br/>nosql connectors<br/>[MongoDBReader.java]")]
  node_hbase_ots[("HBase/OTS<br/>wide-column connectors")]
  node_graph_tsdb[("Graph/TSDB<br/>specialized connectors<br/>[Neo4jWriter.java]")]
  node_cloud_alibaba[("Alibaba Cloud<br/>cloud connectors<br/>[DatahubReader.java]")]
end

subgraph group_ops["Packaging"]
  node_plugin_packaging["Packaging<br/>build/runtime layout<br/>[datax.py]"]
  node_examples["Examples<br/>sample jobs<br/>[Main.java]"]
end

node_engine -->|"reads job"| node_config_parser
node_config_parser -->|"creates"| node_job_container
node_job_container -->|"selects mode"| node_scheduler
node_scheduler -->|"launches"| node_task_group
node_task_group -->|"runs"| node_transport
node_transport -->|"buffers"| node_memory_channel
node_transport -->|"applies"| node_transformers
node_transport -->|"reports"| node_statistics
node_spi_common -->|"contracts"| node_engine
node_spi_common -->|"task API"| node_task_group
node_rdbms_util -->|"shared by"| node_mysql_rdbms
node_rdbms_util -->|"shared by"| node_cloud_alibaba
node_storage_util -->|"shared by"| node_hdfs_oss
node_storage_util -->|"shared by"| node_cloud_alibaba
node_spi_common -->|"implements"| node_mysql_rdbms
node_spi_common -->|"implements"| node_clickhouse
node_spi_common -->|"implements"| node_hdfs_oss
node_spi_common -->|"implements"| node_mongo
node_spi_common -->|"implements"| node_hbase_ots
node_spi_common -->|"implements"| node_graph_tsdb
node_spi_common -->|"implements"| node_cloud_alibaba
node_plugin_packaging -->|"assembles"| node_engine
node_plugin_packaging -->|"packages"| node_mysql_rdbms
node_plugin_packaging -->|"packages"| node_hdfs_oss
node_plugin_packaging -->|"packages"| node_cloud_alibaba
node_examples -->|"demonstrates"| node_engine
node_examples -->|"demonstrates"| node_graph_tsdb

click node_engine "https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/Engine.java"
click node_config_parser "https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java"
click node_job_container "https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java"
click node_scheduler "https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/job/scheduler"
click node_task_group "https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/taskgroup"
click node_transport "https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/transport"
click node_memory_channel "https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java"
click node_transformers "https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/transport/transformer"
click node_statistics "https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/statistics"
click node_spi_common "https://github.com/alibaba/datax/tree/master/common/src/main/java/com/alibaba/datax/common"
click node_rdbms_util "https://github.com/alibaba/datax/tree/master/plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms"
click node_storage_util "https://github.com/alibaba/datax/tree/master/plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage"
click node_mysql_rdbms "https://github.com/alibaba/datax/blob/master/mysqlreader/src/main/java/com/alibaba/datax/plugin/reader/mysqlreader/MysqlReader.java"
click node_clickhouse "https://github.com/alibaba/datax/blob/master/clickhousereader/src/main/java/com/alibaba/datax/plugin/reader/clickhousereader/ClickhouseReader.java"
click node_hdfs_oss "https://github.com/alibaba/datax/blob/master/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/HdfsReader.java"
click node_mongo "https://github.com/alibaba/datax/blob/master/mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/MongoDBReader.java"
click node_hbase_ots "https://github.com/alibaba/datax/blob/master/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xReader.java"
click node_graph_tsdb "https://github.com/alibaba/datax/blob/master/neo4jwriter/src/main/java/com/alibaba/datax/plugin/writer/neo4jwriter/Neo4jWriter.java"
click node_cloud_alibaba "https://github.com/alibaba/datax/blob/master/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubReader.java"
click node_plugin_packaging "https://github.com/alibaba/datax/blob/master/core/src/main/bin/datax.py"
click node_examples "https://github.com/alibaba/datax/blob/master/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_config_parser,node_job_container,node_scheduler,node_task_group,node_transport,node_memory_channel,node_transformers,node_statistics toneBlue
class node_spi_common toneAmber
class node_rdbms_util,node_storage_util toneMint
class node_mysql_rdbms,node_clickhouse,node_hdfs_oss,node_mongo,node_hbase_ots,node_graph_tsdb,node_cloud_alibaba toneRose
class node_plugin_packaging,node_examples toneIndigo
```

## Components

### Core runtime

Job orchestration, scheduling, transport, transformers, and statistics.

- [Engine](https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/Engine.java): job entrypoint · Starts a DataX job and hands control into the core runtime. · `core/src/main/java/com/alibaba/datax/core/Engine.java`
- [Config parser](https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java): job config · Loads and validates job definitions before execution. · `core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java`
- [Job container](https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java): lifecycle coordinator · Coordinates job setup, plugin loading, and lifecycle management. · `core/src/main/java/com/alibaba/datax/core/job/JobContainer.java`
- [Scheduler](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/job/scheduler): job scheduler · Selects standalone or process-inner execution modes for jobs. · `core/src/main/java/com/alibaba/datax/core/job/scheduler`
- [Task group](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/taskgroup): parallel runner · Runs task groups, monitors execution, and manages reader/writer task runners. · `core/src/main/java/com/alibaba/datax/core/taskgroup`
- [Transport](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/transport): record pipeline · Moves records through channels, exchangers, and transformers. · `core/src/main/java/com/alibaba/datax/core/transport`
- [Memory channel](https://github.com/alibaba/datax/blob/master/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java): in-memory channel · Default buffer between reader and writer tasks. · `core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java`
- [Transformers](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/transport/transformer): record transform · Built-in record transformations such as filter, substring, replace, digest, pad, and Groovy. · `core/src/main/java/com/alibaba/datax/core/transport/transformer`
- [Statistics](https://github.com/alibaba/datax/tree/master/core/src/main/java/com/alibaba/datax/core/statistics): metrics/reporting · Collects communication state, reports progress, and emits plugin-level telemetry. · `core/src/main/java/com/alibaba/datax/core/statistics`

### Plugin SPI

Shared plugin contracts and runtime types used by core and connectors.

- [Common SPI](https://github.com/alibaba/datax/tree/master/common/src/main/java/com/alibaba/datax/common): plugin API · Defines Reader/Writer interfaces, record model, exceptions, and plugin base classes. · `common/src/main/java/com/alibaba/datax/common`

### Shared utilities

Reusable connector helpers factored out of individual plugins.

- [RDBMS util](https://github.com/alibaba/datax/tree/master/plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms): shared jdbc lib · Shared JDBC split, metadata, and writer/reader helper logic used by many relational plugins. · `plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms`
- [Storage util](https://github.com/alibaba/datax/tree/master/plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage): shared file lib · Shared file/text/binary/parquet handling and split logic for unstructured storage connectors. · `plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage`

### Connectors

Representative reader and writer plugins across the main integration families.

- [MySQL/PG](https://github.com/alibaba/datax/blob/master/mysqlreader/src/main/java/com/alibaba/datax/plugin/reader/mysqlreader/MysqlReader.java): relational connectors · Representative JDBC reader/writer family for common relational databases. · `mysqlreader/src/main/java/com/alibaba/datax/plugin/reader/mysqlreader/MysqlReader.java`
- [ClickHouse](https://github.com/alibaba/datax/blob/master/clickhousereader/src/main/java/com/alibaba/datax/plugin/reader/clickhousereader/ClickhouseReader.java): warehouse connector · Columnar warehouse connector family. · `clickhousereader/src/main/java/com/alibaba/datax/plugin/reader/clickhousereader/ClickhouseReader.java`
- [HDFS/OSS](https://github.com/alibaba/datax/blob/master/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/HdfsReader.java): file connectors · Distributed storage connectors for file and object storage. · `hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/HdfsReader.java`
- [MongoDB](https://github.com/alibaba/datax/blob/master/mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/MongoDBReader.java): nosql connectors · Document-store reader and writer connectors. · `mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/MongoDBReader.java`
- [HBase/OTS](https://github.com/alibaba/datax/blob/master/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xReader.java): wide-column connectors · HBase and TableStore-style readers/writers with split and mode handling. · `hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xReader.java`
- [Graph/TSDB](https://github.com/alibaba/datax/blob/master/neo4jwriter/src/main/java/com/alibaba/datax/plugin/writer/neo4jwriter/Neo4jWriter.java): specialized connectors · Graph and time-series connector family, including Neo4j, GDB, and TSDB-style integrations. · `neo4jwriter/src/main/java/com/alibaba/datax/plugin/writer/neo4jwriter/Neo4jWriter.java`
- [Alibaba Cloud](https://github.com/alibaba/datax/blob/master/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubReader.java): cloud connectors · DataHub, LogHub, ODPS/MaxCompute, ADS, OCS, and related cloud endpoints. · `datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubReader.java`

### Packaging

Runtime assembly, examples, and distribution artifacts.

- [Packaging](https://github.com/alibaba/datax/blob/master/core/src/main/bin/datax.py): build/runtime layout · Per-plugin assemblies, plugin metadata, bin entrypoints, and distribution scripts. · `core/src/main/bin/datax.py`
- [Examples](https://github.com/alibaba/datax/blob/master/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java): sample jobs · Reference jobs and smoke tests showing framework usage and connector wiring. · `datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java`

## Connections

- Engine → Config parser: reads job · Engine starts by loading and validating job configuration.
- Config parser → Job container: creates · Validated configuration is handed to the job lifecycle coordinator.
- Job container → Scheduler: selects mode · Job container chooses the appropriate execution mode and scheduler.
- Scheduler → Task group: launches · Scheduler dispatches work into parallel task groups.
- Task group → Transport: runs · Task runners use the transport pipeline for record movement.
- Transport → Memory channel: buffers · Records are staged through the default in-memory channel.
- Transport → Transformers: applies · Optional transformers execute between reader output and writer input.
- Transport → Statistics: reports · Transport and task execution update progress and performance data.
- Common SPI → Engine: contracts · Core execution depends on common plugin interfaces and record types.
- Common SPI → Task group: task API · Reader and writer tasks are instantiated through the common SPI.
- RDBMS util → MySQL/PG: shared by · Shared JDBC utilities feed relational reader/writer plugins.
- RDBMS util → Alibaba Cloud: shared by · Some cloud database connectors reuse JDBC-style helper logic.
- Storage util → HDFS/OSS: shared by · Shared file-format and split helpers support storage connectors.
- Storage util → Alibaba Cloud: shared by · Object-storage style connectors reuse unstructured storage helpers.
- Common SPI → MySQL/PG: implements · Relational plugins implement the shared Reader/Writer SPI.
- Common SPI → ClickHouse: implements · Warehouse plugins implement the shared Reader/Writer SPI.
- Common SPI → HDFS/OSS: implements · File and object storage plugins implement the shared Reader/Writer SPI.
- Common SPI → MongoDB: implements · Document-store plugins implement the shared Reader/Writer SPI.
- Common SPI → HBase/OTS: implements · HBase and TableStore plugins implement the shared Reader/Writer SPI.
- Common SPI → Graph/TSDB: implements · Graph and time-series plugins implement the shared Reader/Writer SPI.
- Common SPI → Alibaba Cloud: implements · Alibaba Cloud plugins implement the shared Reader/Writer SPI.
- Packaging → Engine: assembles · The distributed runtime is assembled around the core entrypoint.
- Packaging → MySQL/PG: packages · Each connector is packaged as a separate plugin artifact.
- Packaging → HDFS/OSS: packages · Each connector is packaged as a separate plugin artifact.
- Packaging → Alibaba Cloud: packages · Each connector is packaged as a separate plugin artifact.
- Examples → Engine: demonstrates · Examples show how jobs are launched through the core runtime.
- Examples → Graph/TSDB: demonstrates · Sample jobs exercise representative connector flows.
