How wejorey/datax works
Project type and purpose
• DataX is an open-source, plugin-based offline data synchronization framework.
• It orchestrates “Reader” plugins (to extract records from source data stores) and “Writer” plugins (to load records into targets), core engine components, and utility modules into a unified data-flow pipeline.
Top-level architecture components
a. CLI & entry scripts
• datax.py / dxprof.py / perftrace.py
• Load job JSON, invoke Java Core Engine
b. Core Engine
• Engine.java: main orchestration
• AbstractContainer / JobContainer: per-job lifecycle
• Scheduler (StandAloneScheduler, ProcessInnerScheduler): divides work into TaskGroups
• TaskGroupContainer / Runner (ReaderRunner, WriterRunner): execute plugin tasks
• MemoryChannel: in-process buffering of Record objects
• Transport & Transformer modules: optional record transformations
• Statistics & communication modules (PerfRecord, CommunicationTool)
c. Common library
• SPI interfaces (Reader, Writer, Pluginable, RecordReceiver, RecordSender)
• BaseObject, Configuration, ExceptionTracker, element types (Column, Record)
d. Plugin loader & modules
• plugin.json describes plugin type and class entrypoints
• plugin-rdbms-util and plugin-unstructured-storage-util: shared code for RDBMS and unstructured storage
• Each plugin (mysqlreader, mysqlwriter, hdfsreader, hdfswriter, etc.) packaged as a self-contained Maven module
• Adapters for dozens of data sources (RDBMS, NoSQL, filesystems, big-data engines)
e. DataXService (optional)
• Web service / REST API layer (dataxservice module) for job submission, monitoring, metadata
f. Examples & user guides
• datax-example modules, documentation under README.md, introduction.md, userGuid.md
Relationships & dataflow
• CLI/script → loads job configuration (JSON) → calls Core Engine
• Core Engine → Plugin Loader → loads Reader & Writer plugin jars
• Engine/Scheduler → splits job into TaskGroups
• TaskGroupContainerRunner → instantiates ReaderRunner & WriterRunner
• ReaderRunner → reads data from source via plugin → wraps records → pushes into MemoryChannel
• WriterRunner → pulls records from MemoryChannel → optionally applies Transformer → calls Writer plugin to write to target
• Statistics collected at each stage, communicated back to Engine for reporting
Architectural patterns & principles
• Plugin architecture via SPI (extensibility point for arbitrary data sources)
• Master/worker pattern: one JobContainer spawns multiple TaskGroupContainers (workers)
• Producer/consumer with in-memory channel for decoupling read/write speeds
• Modular codebase: core, common, plugin utilities, individual plugin modules
• Maven multi-module project for independent plugin packaging