# wgzhao/addax architecture

> Architecture diagram of the GitHub repository wgzhao/addax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wgzhao/addax
- Repository: https://github.com/wgzhao/addax

## Overview

The Addax project is a versatile, open‐source ETL tool that uses a modular, plugin-based architecture. Its purpose is to extract data from a wide range of SQL/NoSQL sources, optionally transform it, and then load it into various targets. The system is designed for high extensibility, allowing new readers and writers (connectors) to be easily added via plugins.

##### Main components
- Plugin System: The “plugin” directory is divided into “reader” and “writer” subdirectories. Each plugin package implements a specific interface (for example, implementing Reader.java or Writer.java) so that new data source connections can be integrated seamlessly.
- Additional Libraries/Modules: The “lib” directory (with modules like addax-rdbms and addax-storage) provides additional supporting functionality. These modules encapsulate datasource-specific utilities (e.g. JDBC related functions) and help abstract storage-level operations.
- Job Configurations and Resources: Job files (JSON configurations in core/src/main/job and docs/assets/jobs) drive the ETL operations. There are also resource files (e.g. conf/core.json, logback.xml) that configure system behavior.
- Supporting Infrastructure: Dockerfile, build scripts (build-docker.sh, build-module.sh), and CI/CD workflows provide deployment and integration support.

##### Relationships and interactions between these components
- The Core Engine acts as the conductor—receiving a JSON-based job configuration, initializing the job container and scheduler, and then loading the appropriate reader and writer plugins.
- The Plugin Manager or Plugin Registry (when using the SPI interfaces defined in “core/spi”) scans the “plugin” directories and loads the required reader/writer implementations.
- Data flows from external sources into the Plugin Reader, moves through the internal channel components, and then is handed off to the Plugin Writer for final output.
- Communication and statistics modules (within “core/statistics” and “core/statistics/communication”) monitor performance, coordinate container-level reporting, and ensure transparent logging.

##### Important architectural patterns/design principles
- Modular/Plugin-Based Architecture: New connectors are added without modifying the core. The design uses interfaces (e.g., Reader and Writer SPI) allowing for loose coupling.
- Abstraction Layers: Core engine abstracts job management and scheduling, decoupling ETL logic from the specifics of any data source.
- Container/Task Group Patterns: With classes such as TaskGroupContainer and TaskGroupRunner, the design supports splitting tasks to run concurrently.
- Separation of Concerns: Core modules, plugin modules, and storage modules are clearly separated, each responsible for distinct functionalities (orchestrating jobs, data connectivity, file I/O or database operations).

##### Relevant technologies, frameworks, or libraries
- Java 8 (and above) forms the foundation, with usage of Java Streams in non-performance-sensitive areas.
- Docker is supported for containerized deployment.
- The project also integrates Python scripts for certain operations (e.g. addax.py and dxprof.py).
- CI/CD workflows use GitHub Actions which feed into a continuous integration pipeline.

- Each main component or layer should have a clearly labeled block:
– Core Engine/Job Scheduler
– Plugin Framework (subdivided into Reader Plugins and Writer Plugins)
– Data Channels/Transport (including memory channels and record exchangers)
– Statistics and Monitoring modules
– Deployment and Infrastructure (Docker, CI/CD, build scripts)
– From job configuration (JSON files) into the Core Engine.
– From the Core Engine to the Plugin Loader, and then broadly splitting into Reader and Writer pipelines.
– The orchestration layer (e.g., rectangles for core engine components),
- Labels for important patterns (e.g., “Plugin SPI”, “Task Group Container”, “Process Inner Scheduler”) to highlight design choices.

##### Be detailed
- Don’t over-simplify; include as many components as possible to capture the essence of how various packages interact.
- For instance, detail the relationships inside the “core” module (e.g., listing Engine, AbstractContainer, JobContainer, Scheduler, etc.), then show how plugins are loaded from the “plugin” directory.
- Mention how “lib” modules support specific types of data sources, while “docs” and “config” directories assist with deployment and user instructions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine Layer
    subgraph "Core Engine Layer"
        CE1["Engine"]:::core
        CC["AbstractContainer"]:::core
        JC["JobContainer"]:::core
        JS["Job Scheduling & Process Inner Scheduler"]:::core
        SPI["SPI Interfaces"]:::core
        Stat["Statistics & Monitoring"]:::core
    end

    %% Plugin Framework Layer
    subgraph "Plugin Framework"
        Reader["Reader Plugins"]:::plugins
        Writer["Writer Plugins"]:::plugins
    end

    %% Data Channels / Transport
    subgraph "Data Channels/Transport"
        Channels["Data Channels"]:::data
    end

    %% Additional Libraries/Modules Layer
    subgraph "Additional Libraries"
        LibRDBMS["addax-rdbms"]:::lib
        LibStorage["addax-storage"]:::lib
    end

    %% Job Configurations & Resources Layer
    subgraph "Job Configurations & Resources"
        JobConf["Job Configurations (JSON)"]:::job
        JobDocs["Job Examples"]:::job
        CoreConf["Core Config (core.json)"]:::job
        Logback["Logback Configuration"]:::job
    end

    %% Deployment & Infrastructure Layer
    subgraph "Deployment & Infrastructure"
        Dockerfile["Dockerfile"]:::infrastructure
        BuildDocker["build-docker.sh"]:::infrastructure
        BuildModule["build-module.sh"]:::infrastructure
        CI["CI/CD Workflows"]:::infrastructure
    end

    %% Relationships between layers
    JobConf -->|"triggers"| CE1
    JobDocs -->|"provides"| CE1
    CoreConf -->|"configures"| CE1
    Logback -->|"configures"| CE1

    CE1 -->|"manages"| CC
    CE1 -->|"manages"| JC
    JC -->|"schedul.es"| JS
    CE1 -->|"loads"| SPI
    SPI -->|"discovers"| Reader
    SPI -->|"discovers"| Writer

    Reader -->|"extracts"| Channels
    Channels -->|"forwards to"| Writer
    CE1 -->|"reports to"| Stat

    LibRDBMS ---|"supports"| CE1
    LibStorage ---|"supports"| CE1

    Dockerfile ---|"deploys with"| CI
    BuildDocker ---|"builds"| CI
    BuildModule ---|"builds"| CI
    CI ---|"integrates"| CE1

    %% Styles
    classDef core fill:#ffef96,stroke:#333,stroke-width:2px;
    classDef plugins fill:#c6e2ff,stroke:#333,stroke-width:2px;
    classDef data fill:#d0f0c0,stroke:#333,stroke-width:2px;
    classDef lib fill:#ffd1dc,stroke:#333,stroke-width:2px;
    classDef job fill:#e6e6fa,stroke:#333,stroke-width:2px;
    classDef infrastructure fill:#f4cccc,stroke:#333,stroke-width:2px;

    %% Click Events for core components and mappings
    click CE1 "https://github.com/wgzhao/addax/blob/master/core/src/main/java/com/wgzhao/addax/core/Engine.java"
    click CC "https://github.com/wgzhao/addax/blob/master/core/src/main/java/com/wgzhao/addax/core/AbstractContainer.java"
    click JC "https://github.com/wgzhao/addax/blob/master/core/src/main/java/com/wgzhao/addax/core/job/JobContainer.java"
    click JS "https://github.com/wgzhao/addax/blob/master/core/src/main/java/com/wgzhao/addax/core/job/scheduler/processinner/ProcessInnerScheduler.java"
    click SPI "https://github.com/wgzhao/addax/tree/master/core/src/main/java/com/wgzhao/addax/core/spi"
    click Stat "https://github.com/wgzhao/addax/tree/master/core/src/main/java/com/wgzhao/addax/core/statistics"
    click Reader "https://github.com/wgzhao/addax/tree/master/plugin/reader"
    click Writer "https://github.com/wgzhao/addax/tree/master/plugin/writer"
    click LibRDBMS "https://github.com/wgzhao/addax/tree/master/lib/addax-rdbms"
    click LibStorage "https://github.com/wgzhao/addax/tree/master/lib/addax-storage"
    click JobConf "https://github.com/wgzhao/addax/tree/master/core/src/main/job"
    click JobDocs "https://github.com/wgzhao/addax/tree/master/docs/assets/jobs"
    click CoreConf "https://github.com/wgzhao/addax/blob/master/core/src/main/conf/core.json"
    click Logback "https://github.com/wgzhao/addax/blob/master/core/src/main/conf/logback.xml"
    click Dockerfile "https://github.com/wgzhao/addax/tree/master/Dockerfile"
    click BuildDocker "https://github.com/wgzhao/addax/blob/master/build-docker.sh"
    click BuildModule "https://github.com/wgzhao/addax/blob/master/build-module.sh"
    click CI "https://github.com/wgzhao/addax/tree/master/.github/workflows"
```
