# apecloud/ape-dts architecture

> Architecture diagram of the GitHub repository apecloud/ape-dts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apecloud/ape-dts
- Repository: https://github.com/apecloud/ape-dts

## Overview

The ape‐dts project is a high‐performance, data migration and synchronization tool written in Rust. Its primary purpose is to support “any-to-any” data transfers, including snapshot and CDC (change data capture) between heterogeneous databases. The project not only provides migration but also built-in mechanisms for validating, revising, and processing data (including via user‐supplied Lua scripts), plus the option to run as an HTTP server for pulling data. Its rich feature set and focus on performance are evidenced by the benchmark documentation and extensive test directories.

##### Main Components
- dt‐main: The entry point of the application (the “main” module).
- dt‐connector: Contains the extraction and sinking logic for various databases (MySQL, PostgreSQL, MongoDB, Redis, etc.). Inside, you see subdirectories for extractors for CDC and snapshot modes, as well as sinker implementations (including ones for Kafka, Starrocks, ClickHouse, Doris, etc.).
- dt‐pipeline: Acts as the glue connecting the extractors and sinkers. This module manages the flow (or “pipeline”) of data between the source and target, and it even supports data processing (e.g., Lua-based transformations).
- dt‐parallelizer: Implements several parallel algorithms tailored to improve performance based on the source, target, and type of task. Several files here indicate different parallelization strategies (e.g., for merging or partitioning tasks).
- dt‐task: A configuration-driven module that instantiates the extractors, sinkers, pipelines, and parallelizers according to the given task definitions.
- dt‐common: Provides shared utilities and common data structures such as configuration parsing, metadata handling, logging, error management, and adapters for different database types.
- dt‐tests: A comprehensive test suite (mostly integration tests) that contains tests for every supported scenario (snapshot, CDC, revision, data check, etc.), including Kubernetes configurations for various versions of databases.

##### Relationships and Interactions
- The dt‐main module invokes components produced by dt‐task based on user-supplied configurations (which likely include specifications for the source and target databases, filtering rules, and processing options).
- The dt‐task module uses components from dt‐connector to extract data from the source and dt‐connector sinkers to push data to the target.
- The dt‐pipeline module connects these extractors and sinkers, ensuring the data flows through any processing (such as Lua script modifications) and applies any routing/filters.
- When high performance is needed, dt‐parallelizer is used by dt‐task or dt‐pipeline to run data migration in parallel, reducing latency and increasing throughput.
- dt‐common is used throughout all modules to ensure consistency in configuration management, logging, metadata processing, and error handling.
- The tests (dt‐tests) simulate various real-world scenarios and stress different parts of the architecture to ensure the integrity and performance of the tool.

##### Architectural Patterns & Design Principles
- Modular and decoupled design: Each module focuses on a specific aspect of the system (entry, connectors, pipelines, parallelization, configuration, common utilities), allowing independent development and testing.
- Plug-and-play architecture: The design supports the dynamic creation of tasks with user-defined configurations and the integration of Lua scripts for data transformation, highlighting extensibility.
- Emphasis on parallel processing: With dt‐parallelizer handling various parallel execution strategies, the system is designed from the ground up for high throughput.
- Clearly delineated responsibilities: dt‐connector handles data ingestion/egress; dt‐pipeline channels the data flow, ensuring that transformation logic is centralized; dt‐task orchestrates the entire process.
- Use of containerization: Presence of Dockerfiles and related CI/CD workflows (under .github/workflows) suggests that deployment and scalability are also design concerns.

– dt-main (Entry Point)
– dt-task (Task Creator/Orchestrator)
– dt-pipeline (Data Pipeline)
– dt-parallelizer (Parallel Execution Engine)
– dt-common (Shared Utilities)
– External Systems (Source Databases, Target Databases, Kafka, HTTP Consumers)
– Optionally include dt-tests as a separate testing environment showing integration flows.
– Show that dt-main initiates dt-task which in turn fetches configurations from dt-common.
– Illustrate how dt-pipeline passes data between extractors and sinkers.
– Indicate that dt-parallelizer optimizes flows between these components.
– Show external systems feeding data into extractors and receiving data from sinkers.

– Start with the configuration and orchestration center (dt-task, fed by dt-common) that sets up the migration pipeline.
– Display the separate processing flow: dt-connector extractors from a source, dt-pipeline’s transformation (including Lua scripts), and dt-connector sinkers to the target.
– Highlight the parallelization layer (dt-parallelizer) that boosts performance.
– Connect these to all supported external systems (multiple databases, message queues, HTTP endpoints).
This detailed, segmented approach will clearly illustrate the modular, extensible, and high-performance architecture behind ape-dts.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Core Modules"
        A["dt-main (Entry Point)"]:::core
        B["dt-task (Task Orchestrator)"]:::core
        C["dt-common (Config & Utilities)"]:::core
        D["dt-connector (Data Connectors)"]:::core
        E["Extractors"]:::core
        F["Sinkers"]:::core
        G["dt-pipeline (Data Pipeline)"]:::core
        H["dt-parallelizer (Parallel Engine)"]:::core
    end

    subgraph "External & Testing"
        I["dt-tests (Integration Tests)"]:::test
        J["Source Databases"]:::external
        K["Target Systems & Messaging"]:::external
    end

    %% Core interactions
    A -->|"initiates"| B
    C -->|"supplies_config"| B
    B -->|"instantiates"| D
    B -->|"orchestrates"| G
    B -->|"uses_parallel"| H
    D -->|"handles"| E
    D -->|"handles"| F
    E -->|"flows_to"| G
    G -->|"routes_to"| F
    H -->|"optimizes"| G

    %% External interactions
    J -->|"feeds_data"| E
    F -->|"delivers_data"| K

    %% Testing interactions
    I -->|"tests"| A
    I -->|"tests"| B
    I -->|"tests"| D
    I -->|"tests"| G
    I -->|"tests"| H

    %% Click Events
    click A "https://github.com/apecloud/ape-dts/blob/main/dt-main/src/main.rs"
    click D "https://github.com/apecloud/ape-dts/tree/main/dt-connector/"
    click G "https://github.com/apecloud/ape-dts/tree/main/dt-pipeline/"
    click H "https://github.com/apecloud/ape-dts/tree/main/dt-parallelizer/"
    click B "https://github.com/apecloud/ape-dts/tree/main/dt-task/"
    click C "https://github.com/apecloud/ape-dts/tree/main/dt-common/"
    click I "https://github.com/apecloud/ape-dts/tree/main/dt-tests/"

    %% Styles
    classDef core fill:#bbf,stroke:#333,stroke-width:2px;
    classDef external fill:#cfc,stroke:#333,stroke-width:2px;
    classDef test fill:#fcc,stroke:#333,stroke-width:2px;
```
