How wgzhao/addax works
The Addax project is a versatile, open‐source ETL tool that uses a modular, plugin-based architecture. Its purpose is to extract data from a wide range of SQL/NoSQL sources, optionally transform it, and then load it into various targets. The system is designed for high extensibility, allowing new readers and writers (connectors) to be easily added via plugins.
Main components
• Plugin System: The “plugin” directory is divided into “reader” and “writer” subdirectories. Each plugin package implements a specific interface (for example, implementing Reader.java or Writer.java) so that new data source connections can be integrated seamlessly.
• Additional Libraries/Modules: The “lib” directory (with modules like addax-rdbms and addax-storage) provides additional supporting functionality. These modules encapsulate datasource-specific utilities (e.g. JDBC related functions) and help abstract storage-level operations.
• Job Configurations and Resources: Job files (JSON configurations in core/src/main/job and docs/assets/jobs) drive the ETL operations. There are also resource files (e.g. conf/core.json, logback.xml) that configure system behavior.
• Supporting Infrastructure: Dockerfile, build scripts (build-docker.sh, build-module.sh), and CI/CD workflows provide deployment and integration support.
Relationships and interactions between these components
• The Core Engine acts as the conductor—receiving a JSON-based job configuration, initializing the job container and scheduler, and then loading the appropriate reader and writer plugins.
• The Plugin Manager or Plugin Registry (when using the SPI interfaces defined in “core/spi”) scans the “plugin” directories and loads the required reader/writer implementations.
• Data flows from external sources into the Plugin Reader, moves through the internal channel components, and then is handed off to the Plugin Writer for final output.
• Communication and statistics modules (within “core/statistics” and “core/statistics/communication”) monitor performance, coordinate container-level reporting, and ensure transparent logging.
Important architectural patterns/design principles
• Modular/Plugin-Based Architecture: New connectors are added without modifying the core. The design uses interfaces (e.g., Reader and Writer SPI) allowing for loose coupling.
• Abstraction Layers: Core engine abstracts job management and scheduling, decoupling ETL logic from the specifics of any data source.
• Container/Task Group Patterns: With classes such as TaskGroupContainer and TaskGroupRunner, the design supports splitting tasks to run concurrently.
• Separation of Concerns: Core modules, plugin modules, and storage modules are clearly separated, each responsible for distinct functionalities (orchestrating jobs, data connectivity, file I/O or database operations).
Relevant technologies, frameworks, or libraries
• Java 8 (and above) forms the foundation, with usage of Java Streams in non-performance-sensitive areas.
• Docker is supported for containerized deployment.
• The project also integrates Python scripts for certain operations (e.g. addax.py and dxprof.py).
• CI/CD workflows use GitHub Actions which feed into a continuous integration pipeline.
• Each main component or layer should have a clearly labeled block:
– Core Engine/Job Scheduler
– Plugin Framework (subdivided into Reader Plugins and Writer Plugins)
– Data Channels/Transport (including memory channels and record exchangers)
– Statistics and Monitoring modules
– Deployment and Infrastructure (Docker, CI/CD, build scripts)
– From job configuration (JSON files) into the Core Engine.
– From the Core Engine to the Plugin Loader, and then broadly splitting into Reader and Writer pipelines.
– The orchestration layer (e.g., rectangles for core engine components),
• Labels for important patterns (e.g., “Plugin SPI”, “Task Group Container”, “Process Inner Scheduler”) to highlight design choices.
Be detailed
• Don’t over-simplify; include as many components as possible to capture the essence of how various packages interact.
• For instance, detail the relationships inside the “core” module (e.g., listing Engine, AbstractContainer, JobContainer, Scheduler, etc.), then show how plugins are loaded from the “plugin” directory.
• Mention how “lib” modules support specific types of data sources, while “docs” and “config” directories assist with deployment and user instructions.