# yugabyte/yb-voyager architecture

> Architecture diagram of the GitHub repository yugabyte/yb-voyager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yugabyte/yb-voyager
- Repository: https://github.com/yugabyte/yb-voyager

## Overview

##### Project Type and Purpose
- Model YugabyteDB Voyager as a full‐stack, CLI-driven migration engine.
- That it handles end-to-end migration: starting from cluster preparation and schema export, through data export/import, to progress and status reporting.
- Point out its support for multiple source database types, integration with cloud storage (S3, GCS, Azure), and native YugabyteDB interactions.

##### Main Components
- Command-Line Interface (CLI Layer): Represented by the “yb-voyager” directory where various commands such as “analyzeSchema”, “exportData”, “importData”, “cutover”, etc. are implemented.
- Core Migration Engine: The primary business logic is split across packages that manage configuration (src/config), orchestrate the migration process (src/migassessment, src/cp/control_plane, etc.), and handle state tracking (metadata in src/metadb).
- Data Connectors and Adapters: Distinct modules to handle connecting to sources (src/srcdb/oracle.go, src/srcdb/postgres.go, src/srcdb/mysql.go) and targets (src/tgtdb/).
- Data File Management: Components under src/datafile handle import/export of CSV, text, and other file formats.
- Query and Transformation: The packages under src/query (including queryparser, sqltransformer) represent components for analyzing schemas, transforming queries, and detecting issues.
- Reporting and Progress Tracking: Modules under src/reporter and src/migassessment generate reports and progress metadata.
- CDC Integration: The “debezium-server-voyager” directories (and the exporter package) encapsulate capabilities such as change-data-capture processing.
- Utility and Integration Modules: Various utilities (like src/utils, src/namereg) and the callhome component for diagnostic reporting.
- Testing & Migration Tests: The “migtests” directory holds extensive integration and schema migration tests indicating that robustness and automation testing are key parts of the architecture.
- Auxiliary Components: Installer scripts, docker deployment definitions, and CI/CD definitions (in .github/workflows) to manage build, testing, and deployment.

##### Relationships and Interactions
- That data, once exported, flow into transformation modules (sqltransformer, queryparser) and then to the Data File module (datafile) for downstream import.
- A component for CDC/streaming using the Debezium-based modules and show how they may produce events that feed into the migration logic.
- From the migration engine, show an output path into a Reporting module that generates progress updates and migration status, feeding back into the CLI or external dashboards.

##### Important Architectural Patterns and Design Principles
- Modularity: each functional module (metadata, connectors, transformation, reporting) is kept in its own package.
- Use adapter/inversion-of-control pattern for interacting with different data sources and targets.
- Separation between control-plane (orchestration logic in src/cp) and data-plane (actual data exports, transformation, and import).
- Scalability and parallelism aspects mentioned in the README (e.g., adaptive parallelism in src/adaptiveparallelism).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLI["yb-voyager (CLI)"]:::cli
    end

    %% Core Migration Engine
    subgraph "Core Migration Engine"
        CFG["Configuration"]:::core
        CP["Control Plane"]:::core
        META["Metadata/State Tracking"]:::core
        MIG["Migration Assessment"]:::core
    end

    %% Data Connectors and Adapters
    subgraph "Data Connectors & Adapters"
        SRC["Source Connectors"]:::connectors
        TGT["Target Connectors"]:::connectors
    end

    %% Query and Transformation
    subgraph "Query & Transformation"
        QP["Query Parsing"]:::transformation
        SQLT["SQL Transformation"]:::transformation
    end

    %% Data File Management
    subgraph "Data File Management"
        DATAFILE["Data File Manager"]:::data
    end

    %% Reporting and Progress Tracking
    subgraph "Reporting & Progress"
        REP["Reporting & Status"]:::report
    end

    %% CDC Integration
    subgraph "CDC Integration"
        CDC_SRV["Debezium Server"]:::cdc
        CDC_EXP["CDC Exporter"]:::cdc
        CDC_CLI["CDC Client Wrapper"]:::cdc
    end

    %% Utility and Integration Modules
    subgraph "Utilities & Integrations"
        UTIL["Utilities"]:::util
        STORE["Cloud Storage Connectors"]:::util
    end

    %% Testing & CI/CD
    subgraph "Testing & CI/CD"
        TESTS["Migration Tests"]:::test
        CICD["CI/CD Workflows"]:::test
    end

    %% Relationships and Data Flow
    CLI -->|"initiate"| CFG
    CLI --> CP
    CLI --> META
    CLI --> MIG

    CFG -->|"load_settings"| CP
    CP --> SRC
    CP --> TGT
    CP --> QP
    CP --> SQLT
    CP --> DATAFILE
    CP --> REP

    SRC -->|"export_data"| QP
    TGT -->|"prepare_import"| DATAFILE
    QP --> SQLT
    SQLT --> DATAFILE
    DATAFILE -->|"feed_import"| TGT

    CDC_SRV --> CDC_EXP
    CDC_EXP --> CDC_CLI
    CDC_CLI --> CP

    REP -->|"report_status"| CLI

    UTIL --> CP
    STORE --> CP

    TESTS -->|"validate_process"| CLI
    CICD -->|"build_deploy"| CLI

    %% Click Events
    click CLI "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/cmd"
    click CFG "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/config"
    click CP "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/cp"
    click META "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/metadb"
    click MIG "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/migassessment"
    click SRC "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/srcdb"
    click TGT "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/tgtdb"
    click DATAFILE "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/datafile"
    click QP "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/query/queryparser"
    click SQLT "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/query/sqltransformer"
    click REP "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/reporter"
    click CDC_SRV "https://github.com/yugabyte/yb-voyager/tree/main/debezium-server-voyager"
    click CDC_EXP "https://github.com/yugabyte/yb-voyager/tree/main/debezium-server-voyager/debezium-server-voyagerexporter"
    click CDC_CLI "https://github.com/yugabyte/yb-voyager/tree/main/debezium-server-voyager/yb-client-cdc-stream-wrapper"
    click UTIL "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/utils"
    click STORE "https://github.com/yugabyte/yb-voyager/tree/main/yb-voyager/src/datastore"
    click TESTS "https://github.com/yugabyte/yb-voyager/tree/main/migtests"

    %% Styles
    classDef cli fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef core fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef connectors fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef transformation fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef data fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef report fill:#D8BFD8,stroke:#333,stroke-width:2px;
    classDef cdc fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef util fill:#F0E68C,stroke:#333,stroke-width:2px;
    classDef test fill:#E6E6FA,stroke:#333,stroke-width:2px;
```
