How alibaba/datax works
1. What this repository is
- This is the open-source DataX data integration platform: a Java/Maven multi-module ETL framework plus a large catalog of reader/writer plugins.
- The repo is organized around a core execution engine and many connector modules for databases, warehouses, filesystems, graph stores, time-series systems, and Alibaba Cloud services.
2. Top-level architecture
- The system has three main layers:
- Core runtime: parses jobs, schedules work, runs tasks, tracks metrics.
- Plugin SPI: standardized Reader/Writer interfaces and shared record/exception/config classes.
- Connector implementations: one module per source/target system.
- Architecture diagrams should show DataX as a job orchestration engine that sits between a source system and a sink system, with optional transformation steps in the middle.
3. Core execution engine
- The core module contains the orchestration path:
- Engine starts a job.
- ConfigParser and related utilities load/validate job definitions.
- JobContainer coordinates job lifecycle.
- TaskGroupContainer runs parallel task groups and TaskMonitor supervises execution.
- Scheduling is built around standalone/process-inner modes, with communication and reporting components under core/statistics.
4. Data movement model
- Data flows through an internal transport pipeline:
- Readers produce Record objects.
- Records travel through an in-memory Channel implementation.
- Writers consume records from the channel.
- The transport layer is in core/transport, and the default channel is MemoryChannel.
- This means the architecture is push/pull between task runners, not a direct source-to-sink call path.
5. Plugin contract and shared runtime types
- The common module defines the plugin API surface:
- Reader / Writer SPI interfaces.
- AbstractJobPlugin / AbstractTaskPlugin bases.
- Record, Column types, and casting utilities.
- Central exceptions, error codes, configuration helpers, and plugin collectors.
- common is the main boundary between the core engine and connector modules.
6. Transformation subsystem
- DataX supports record-level transformations through the transformer module and the transformer implementations in core/transport/transformer.
- Built-in transformers include filtering, substring, replace, pad, digest, and Groovy-based transformation.
- In the architecture, transformers sit between reader output and writer input, inside the transport/exchanger pipeline.
7. Connector/plugin modules
- Most directories at the repo root are standalone plugin modules, each packaging one source or sink integration.
- There are broad families:
- RDBMS: MySQL, Oracle, PostgreSQL, SQL Server, DRDS, Kingbase, OceanBase, GaussDB, etc.
- Warehouses/lake/storage: ODPS/MaxCompute, HDFS/Hive, OSS, FTP, ClickHouse, Doris, StarRocks, Databend, SelectDB, Hologres.
- NoSQL/graph/time-series: MongoDB, Cassandra, HBase variants, OTS, GDB, Neo4j, TDengine, TSDB, OpenTSDB.
- Alibaba Cloud middleware: DataHub, LogHub, OCS, ADS, ADB, SLS.
- Each plugin module usually contains:
- a main Reader/Writer class,
- a Key or Constant config holder,
- error codes,
- helper/utility classes for splitting, mapping, format conversion, or remote client access,
- plugin.json and a job template.
8. Shared connector utilities
- Some subsystems are factored out and reused by multiple plugins:
- plugin-rdbms-util for common JDBC split, metadata, and DB helper logic.
- plugin-unstructured-storage-util for file/text/binary/parquet handling and split logic.
- These modules reduce duplication across many readers/writers and should be drawn as shared libraries feeding several connector modules.
9. Plugin packaging and deployment
- Each plugin module has its own pom.xml and src/main/assembly/package.xml, indicating it is built and packaged separately.
- Runtime plugin metadata is carried in plugin.json and plugin_job_template.json.
- The core/src/main/bin/datax.py entrypoint and core/src/main/conf files indicate the distributed runtime is assembled as a runnable DataX installation with plugins dropped into a plugin directory structure.
10. Examples, tests, and operational artifacts
- datax-example contains sample jobs and example code showing how the framework is invoked.
- Many plugin modules include unit/integration tests under src/test.
- rpm/ contains packaging scripts for system-level distribution.
- These are not part of the runtime data path, but they document and validate the architecture.
11. Important architectural boundary
- The key boundary in this repo is between:
- framework concerns: parsing, scheduling, concurrency, transport, metrics, error handling;
- connector concerns: protocol specifics, schema mapping, batching, partitioning, type conversion, and remote service clients.
- An architecture diagram should keep the core engine centralized and show each plugin module as a pluggable endpoint using the common SPI.
12. How to read the whole repo
- Think of the repository as a plugin ecosystem around a single batch data pipeline:
- job definition enters the core,
- core loads matching reader/writer plugins,
- reader tasks fetch and split source data,
- records flow through memory and optional transformers,
- writer tasks persist to the target system,
- statistics/reporting track the run.
Main entry points: common Common SPI · datax.py Packaging · transport Transport · Engine.java Engine
Components 21
Core runtime
Job orchestration, scheduling, transport, transformers, and statistics.
- Engine · Starts a DataX job and hands control into the core runtime.
core/src/main/java/com/alibaba/datax/core/Engine.java - Config parser · Loads and validates job definitions before execution.
core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java - Job container · Coordinates job setup, plugin loading, and lifecycle management.
core/src/main/java/com/alibaba/datax/core/job/JobContainer.java - Scheduler · Selects standalone or process-inner execution modes for jobs.
core/src/main/java/com/alibaba/datax/core/job/scheduler - Task group · Runs task groups, monitors execution, and manages reader/writer task runners.
core/src/main/java/com/alibaba/datax/core/taskgroup - Transport · Moves records through channels, exchangers, and transformers.
core/src/main/java/com/alibaba/datax/core/transport - Memory channel · Default buffer between reader and writer tasks.
core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java - Transformers · Built-in record transformations such as filter, substring, replace, digest, pad, and Groovy.
core/src/main/java/com/alibaba/datax/core/transport/transformer - Statistics · Collects communication state, reports progress, and emits plugin-level telemetry.
core/src/main/java/com/alibaba/datax/core/statistics
Plugin SPI
Shared plugin contracts and runtime types used by core and connectors.
- Common SPI · Defines Reader/Writer interfaces, record model, exceptions, and plugin base classes.
common/src/main/java/com/alibaba/datax/common
Shared utilities
Reusable connector helpers factored out of individual plugins.
- RDBMS util · Shared JDBC split, metadata, and writer/reader helper logic used by many relational plugins.
plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms - Storage util · Shared file/text/binary/parquet handling and split logic for unstructured storage connectors.
plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage
Connectors
Representative reader and writer plugins across the main integration families.
- MySQL/PG · Representative JDBC reader/writer family for common relational databases.
mysqlreader/src/main/java/com/alibaba/datax/plugin/reader/mysqlreader/MysqlReader.java - ClickHouse · Columnar warehouse connector family.
clickhousereader/src/main/java/com/alibaba/datax/plugin/reader/clickhousereader/ClickhouseReader.java - HDFS/OSS · Distributed storage connectors for file and object storage.
hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/HdfsReader.java - MongoDB · Document-store reader and writer connectors.
mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/MongoDBReader.java - HBase/OTS · HBase and TableStore-style readers/writers with split and mode handling.
hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xReader.java - Graph/TSDB · Graph and time-series connector family, including Neo4j, GDB, and TSDB-style integrations.
neo4jwriter/src/main/java/com/alibaba/datax/plugin/writer/neo4jwriter/Neo4jWriter.java - Alibaba Cloud · DataHub, LogHub, ODPS/MaxCompute, ADS, OCS, and related cloud endpoints.
datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubReader.java
Packaging
Runtime assembly, examples, and distribution artifacts.
Connections 27
- Engine → Config parser: reads job · Engine starts by loading and validating job configuration.
- Config parser → Job container: creates · Validated configuration is handed to the job lifecycle coordinator.
- Job container → Scheduler: selects mode · Job container chooses the appropriate execution mode and scheduler.
- Scheduler → Task group: launches · Scheduler dispatches work into parallel task groups.
- Task group → Transport: runs · Task runners use the transport pipeline for record movement.
- Transport → Memory channel: buffers · Records are staged through the default in-memory channel.
- Transport → Transformers: applies · Optional transformers execute between reader output and writer input.
- Transport → Statistics: reports · Transport and task execution update progress and performance data.
- Common SPI → Engine: contracts · Core execution depends on common plugin interfaces and record types.
- Common SPI → Task group: task API · Reader and writer tasks are instantiated through the common SPI.
- RDBMS util → MySQL/PG: shared by · Shared JDBC utilities feed relational reader/writer plugins.
- RDBMS util → Alibaba Cloud: shared by · Some cloud database connectors reuse JDBC-style helper logic.
- Storage util → HDFS/OSS: shared by · Shared file-format and split helpers support storage connectors.
- Storage util → Alibaba Cloud: shared by · Object-storage style connectors reuse unstructured storage helpers.
- Common SPI → MySQL/PG: implements · Relational plugins implement the shared Reader/Writer SPI.
- Common SPI → ClickHouse: implements · Warehouse plugins implement the shared Reader/Writer SPI.
- Common SPI → HDFS/OSS: implements · File and object storage plugins implement the shared Reader/Writer SPI.
- Common SPI → MongoDB: implements · Document-store plugins implement the shared Reader/Writer SPI.
- Common SPI → HBase/OTS: implements · HBase and TableStore plugins implement the shared Reader/Writer SPI.
- Common SPI → Graph/TSDB: implements · Graph and time-series plugins implement the shared Reader/Writer SPI.
- Common SPI → Alibaba Cloud: implements · Alibaba Cloud plugins implement the shared Reader/Writer SPI.
- Packaging → Engine: assembles · The distributed runtime is assembled around the core entrypoint.
- Packaging → MySQL/PG: packages · Each connector is packaged as a separate plugin artifact.
- Packaging → HDFS/OSS: packages · Each connector is packaged as a separate plugin artifact.
- Packaging → Alibaba Cloud: packages · Each connector is packaged as a separate plugin artifact.
- Examples → Engine: demonstrates · Examples show how jobs are launched through the core runtime.
- Examples → Graph/TSDB: demonstrates · Sample jobs exercise representative connector flows.