How yugabyte/yb-voyager works
Project Type and Purpose
• Model YugabyteDB Voyager as a full‐stack, CLI-driven migration engine.
• That it handles end-to-end migration: starting from cluster preparation and schema export, through data export/import, to progress and status reporting.
• Point out its support for multiple source database types, integration with cloud storage (S3, GCS, Azure), and native YugabyteDB interactions.
Main Components
• Command-Line Interface (CLI Layer): Represented by the “yb-voyager” directory where various commands such as “analyzeSchema”, “exportData”, “importData”, “cutover”, etc. are implemented.
• Core Migration Engine: The primary business logic is split across packages that manage configuration (src/config), orchestrate the migration process (src/migassessment, src/cp/control_plane, etc.), and handle state tracking (metadata in src/metadb).
• Data Connectors and Adapters: Distinct modules to handle connecting to sources (src/srcdb/oracle.go, src/srcdb/postgres.go, src/srcdb/mysql.go) and targets (src/tgtdb/).
• Data File Management: Components under src/datafile handle import/export of CSV, text, and other file formats.
• Query and Transformation: The packages under src/query (including queryparser, sqltransformer) represent components for analyzing schemas, transforming queries, and detecting issues.
• Reporting and Progress Tracking: Modules under src/reporter and src/migassessment generate reports and progress metadata.
• CDC Integration: The “debezium-server-voyager” directories (and the exporter package) encapsulate capabilities such as change-data-capture processing.
• Utility and Integration Modules: Various utilities (like src/utils, src/namereg) and the callhome component for diagnostic reporting.
• Testing & Migration Tests: The “migtests” directory holds extensive integration and schema migration tests indicating that robustness and automation testing are key parts of the architecture.
• Auxiliary Components: Installer scripts, docker deployment definitions, and CI/CD definitions (in .github/workflows) to manage build, testing, and deployment.
Relationships and Interactions
• That data, once exported, flow into transformation modules (sqltransformer, queryparser) and then to the Data File module (datafile) for downstream import.
• A component for CDC/streaming using the Debezium-based modules and show how they may produce events that feed into the migration logic.
• From the migration engine, show an output path into a Reporting module that generates progress updates and migration status, feeding back into the CLI or external dashboards.
Important Architectural Patterns and Design Principles
• Modularity: each functional module (metadata, connectors, transformation, reporting) is kept in its own package.
• Use adapter/inversion-of-control pattern for interacting with different data sources and targets.
• Separation between control-plane (orchestration logic in src/cp) and data-plane (actual data exports, transformation, and import).
• Scalability and parallelism aspects mentioned in the README (e.g., adaptive parallelism in src/adaptiveparallelism).