# wejorey/datax architecture

> Architecture diagram of the GitHub repository wejorey/datax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wejorey/datax
- Repository: https://github.com/wejorey/datax

## Overview

##### Project type and purpose
- DataX is an open-source, plugin-based offline data synchronization framework.
- It orchestrates “Reader” plugins (to extract records from source data stores) and “Writer” plugins (to load records into targets), core engine components, and utility modules into a unified data-flow pipeline.

##### Top-level architecture components
a. CLI & entry scripts
- datax.py / dxprof.py / perftrace.py
- Load job JSON, invoke Java Core Engine
b. Core Engine
- Engine.java: main orchestration
- AbstractContainer / JobContainer: per-job lifecycle
- Scheduler (StandAloneScheduler, ProcessInnerScheduler): divides work into TaskGroups
- TaskGroupContainer / Runner (ReaderRunner, WriterRunner): execute plugin tasks
- MemoryChannel: in-process buffering of Record objects
- Transport & Transformer modules: optional record transformations
- Statistics & communication modules (PerfRecord, CommunicationTool)
c. Common library
- SPI interfaces (Reader, Writer, Pluginable, RecordReceiver, RecordSender)
- BaseObject, Configuration, ExceptionTracker, element types (Column, Record)
d. Plugin loader & modules
- plugin.json describes plugin type and class entrypoints
- plugin-rdbms-util and plugin-unstructured-storage-util: shared code for RDBMS and unstructured storage
- Each plugin (mysqlreader, mysqlwriter, hdfsreader, hdfswriter, etc.) packaged as a self-contained Maven module
- Adapters for dozens of data sources (RDBMS, NoSQL, filesystems, big-data engines)
e. DataXService (optional)
- Web service / REST API layer (dataxservice module) for job submission, monitoring, metadata
f. Examples & user guides
- datax-example modules, documentation under README.md, introduction.md, userGuid.md

##### Relationships & dataflow
- CLI/script → loads job configuration (JSON) → calls Core Engine
- Core Engine → Plugin Loader → loads Reader & Writer plugin jars
- Engine/Scheduler → splits job into TaskGroups
- TaskGroupContainerRunner → instantiates ReaderRunner & WriterRunner
- ReaderRunner → reads data from source via plugin → wraps records → pushes into MemoryChannel
- WriterRunner → pulls records from MemoryChannel → optionally applies Transformer → calls Writer plugin to write to target
- Statistics collected at each stage, communicated back to Engine for reporting

##### Architectural patterns & principles
- Plugin architecture via SPI (extensibility point for arbitrary data sources)
- Master/worker pattern: one JobContainer spawns multiple TaskGroupContainers (workers)
- Producer/consumer with in-memory channel for decoupling read/write speeds
- Modular codebase: core, common, plugin utilities, individual plugin modules
- Maven multi-module project for independent plugin packaging

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Entry
    subgraph "CLI / Entry Scripts"
        direction TB
        CLI_datax["datax.py"]:::external
        CLI_dxprof["dxprof.py"]:::external
        CLI_perf["perftrace.py"]:::external
    end

    %% Core Engine
    subgraph "Core Engine" 
        direction TB
        Engine["Engine.java"]:::core
        AbstractCont["AbstractContainer.java"]:::core
        JobCont["JobContainer.java"]:::core
        subgraph "Scheduler"
            direction TB
            AbstractSched["AbstractScheduler.java"]:::core
            StandAloneSched["StandAloneScheduler.java"]:::core
            ProcessInnerSched["ProcessInnerScheduler.java"]:::core
        end
        subgraph "TaskGroup & Runners"
            direction TB
            TGContainer["TaskGroupContainer.java"]:::core
            TGRunner["TaskGroupContainerRunner.java"]:::core
            ReaderRun["ReaderRunner.java"]:::core
            WriterRun["WriterRunner.java"]:::core
        end
        MemoryChan["MemoryChannel.java"]:::core
        subgraph "Transport & Transformer"
            direction TB
            Exchanger["RecordExchanger / TransformerExchanger"]:::core
            Transformer["Transformer modules"]:::core
        end
        subgraph "Statistics & Communication"
            direction TB
            CommTool["CommunicationTool.java"]:::core
            PerfRec["PerfRecord.java"]:::common
        end
    end

    %% Common SPI & Utilities
    subgraph "Common SPI & Utilities"
        direction TB
        SPIReader["Reader.java"]:::common
        SPIWriter["Writer.java"]:::common
        BaseObj["BaseObject.java"]:::common
        Config["Configuration.java"]:::common
        ExTracker["ExceptionTracker.java"]:::common
        RecordElm["Record.java"]:::common
        ColumnElm["Column.java"]:::common
    end

    %% Plugin Loader & Shared Utils
    subgraph "Plugin Loader & Shared Utilities"
        direction TB
        CoreConst["CoreConstant.java"]:::core
        JarLoad["JarLoader.java"]:::core
        RdbmsUtil["plugin-rdbms-util"]:::utils
        UnstructUtil["plugin-unstructured-storage-util"]:::utils
    end

    %% Plugin Ecosystem
    subgraph "Plugin Ecosystem"
        direction TB
        subgraph "Example MySQL Plugin"
            direction TB
            MySQLReader["mysqlreader plugin.json & MysqlReader.java"]:::plugin
            MySQLWriter["mysqlwriter plugin.json & MysqlWriter.java"]:::plugin
        end
        OtherPlugins["... other adapters"]:::plugin
    end

    %% Optional Service
    DataXService["DataXService (REST API)"]:::external

    %% Examples & Docs
    subgraph "Examples & Documentation"
        direction TB
        ExampleCore["Main.java (datax-example)"]:::external
        Docs["README.md / introduction.md / userGuid.md"]:::external
    end

    %% Flow
    CLI_datax -->|"loads job JSON"| Engine
    CLI_dxprof --> Engine
    CLI_perf --> Engine

    Engine -->|"uses SPI & Config"| SPIReader
    Engine --> SPIWriter
    Engine --> Config
    Engine --> AbstractCont
    AbstractCont --> JobCont
    JobCont --> AbstractSched
    AbstractSched --> StandAloneSched
    AbstractSched --> ProcessInnerSched
    JobCont --> TGContainer
    TGContainer --> TGRunner
    TGRunner --> ReaderRun
    TGRunner --> WriterRun

    ReaderRun -->|"read records"| MemoryChan
    MemoryChan -->|"buffer records"| WriterRun
    WriterRun -->|"write records"| WriterRun

    ReaderRun --> Exchanger
    WriterRun --> Exchanger
    Exchanger --> Transformer

    ReaderRun --> CommTool
    WriterRun --> CommTool
    CommTool --> PerfRec

    Engine --> CoreConst
    Engine --> JarLoad
    CoreConst --> RdbmsUtil
    CoreConst --> UnstructUtil

    JarLoad --> MySQLReader
    JarLoad --> MySQLWriter
    JarLoad --> OtherPlugins

    DataXService --> JobCont
    Engine --> DataXService

    ExampleCore --> Docs

    %% Click Events
    click CLI_datax "https://github.com/wejorey/datax/blob/master/core/src/main/bin/datax.py"
    click CLI_dxprof "https://github.com/wejorey/datax/blob/master/core/src/main/bin/dxprof.py"
    click CLI_perf "https://github.com/wejorey/datax/blob/master/core/src/main/bin/perftrace.py"
    click Engine "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/Engine.java"
    click AbstractCont "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/AbstractContainer.java"
    click JobCont "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java"
    click TGContainer "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/taskgroup/TaskGroupContainer.java"
    click AbstractSched "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/scheduler/AbstractScheduler.java"
    click StandAloneSched "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/scheduler/processinner/StandAloneScheduler.java"
    click ProcessInnerSched "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/job/scheduler/processinner/ProcessInnerScheduler.java"
    click TGRunner "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/taskgroup/runner/TaskGroupContainerRunner.java"
    click ReaderRun "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/taskgroup/runner/ReaderRunner.java"
    click WriterRun "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/taskgroup/runner/WriterRunner.java"
    click MemoryChan "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java"
    click CommTool "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/statistics/communication/CommunicationTool.java"
    click PerfRec "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/statistics/PerfRecord.java"
    click SPIReader "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/spi/Reader.java"
    click SPIWriter "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/spi/Writer.java"
    click BaseObj "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/base/BaseObject.java"
    click Config "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/util/Configuration.java"
    click ExTracker "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/exception/ExceptionTracker.java"
    click RecordElm "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/element/Record.java"
    click ColumnElm "https://github.com/wejorey/datax/blob/master/common/src/main/java/com/alibaba/datax/common/element/Column.java"
    click CoreConst "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/util/container/CoreConstant.java"
    click JarLoad "https://github.com/wejorey/datax/blob/master/core/src/main/java/com/alibaba/datax/core/util/container/JarLoader.java"
    click RdbmsUtil "https://github.com/wejorey/datax/tree/master/plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms/"
    click UnstructUtil "https://github.com/wejorey/datax/tree/master/plugin-unstructured-storage-util/src/main/java/com/alibaba/datax/plugin/unstructuredstorage/"
    click MySQLReader "https://github.com/wejorey/datax/blob/master/mysqlreader/src/main/resources/plugin.json"
    click MySQLReader "https://github.com/wejorey/datax/blob/master/mysqlreader/src/main/java/com/alibaba/datax/plugin/reader/mysqlreader/MysqlReader.java"
    click MySQLWriter "https://github.com/wejorey/datax/blob/master/mysqlwriter/src/main/resources/plugin.json"
    click MySQLWriter "https://github.com/wejorey/datax/blob/master/mysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/mysqlwriter/MysqlWriter.java"

    %% Styles
    classDef core fill:#D0E6FF,stroke:#0366d6,color:#0366d6;
    classDef common fill:#F0F0F0,stroke:#555,color:#333;
    classDef plugin fill:#E6FFE6,stroke:#28A745,color:#155724;
    classDef utils fill:#FFF8DC,stroke:#DAA520,color:#8B4513;
    classDef external fill:#FFEFD5,stroke:#FF8C00,color:#CD6600;
```
