How udaythakare/migration-tool works
1. Two-tier architecture
- The repo is split into a Next.js frontend under frontend/ and a Python migration engine under server/.
- The frontend is the operator console; the Python side performs schema analysis, mapping, and data movement.
2. Frontend: interactive migration workspace
- frontend/app/page.js is the main dashboard entry point.
- It handles schema import, mapping generation, workspace URL state, and local relationship parsing for table graphs.
- TableMappingCard.js and TableMappingDetail.js are the main presentation/editing components for table-level and column-level mapping review.
3. Frontend API layer as a control boundary
- frontend/app/api/... acts as the bridge between the UI and backend engine.
- The routes are organized by domain:
- api/engine/[action]/route.js for invoking migration engine actions
- api/mappings/route.js for saving/loading mapping state
- api/projects/route.js for project/workspace management
- This layer keeps the browser UI decoupled from direct Python execution.
4. Python engine orchestration
- server/complete_migration.py is the top-level orchestrator for running a migration end to end.
- It likely coordinates extraction, mapping, ID remapping, transformation, and execution order.
- server/main.py is the central service entrypoint for backend APIs or command-driven execution.
5. Schema extraction and parsing
- server/mysql_extractor.py handles live source schema extraction from MySQL.
- server/data_parser.py and server/data_transformers.py normalize incoming table/column data and prepare it for migration.
- server/parse_legacy_schema is also described in the README as the upload path for .sql schema files used by the UI.
6. Mapping engine: deterministic first, AI-assisted second
- server/hybrid_mapper.py appears to be the main mapping strategy, combining heuristics with model-based assistance.
- server/simple_mapper.py and server/ai_schema_mapper.py suggest a fallback/alternate mapping pipeline.
- server/gemini_schema_mapper.py and server/groq_schema_mapper.py represent model-specific schema matching backends.
- The architecture is centered on producing table/column correspondences between source MySQL and target PostgreSQL schemas.
7. Dependency ordering and conflict handling
- The migration is not a flat copy; it needs a safe execution order.
- The README and filenames indicate special handling for parent/child table ordering, circular dependencies, and conflict resolution.
- server/id_remapper.py handles primary key and foreign key translation so relational integrity survives the move.
8. Remote target execution
- server/connect_vps.py is the remote infrastructure boundary for connecting to a VPS-hosted target database.
- The README mentions secure remote access and retry/timeout behavior, so the engine is designed to operate against non-local infrastructure.
- This is an important boundary because migration execution depends on networked database access, not just local files.
9. Persistent mapping and migration artifacts
- Mapping state is stored in JSON artifacts such as server/mapping.json and server/mappings/v1.0.0.json.
- The frontend also has .local-data/migration-store.json, which suggests local workspace persistence for the UI.
- These artifacts are the handoff point between planning in the UI and execution in the Python engine.
10. Operational inputs and test fixtures
- The SQL files (server/lagacy.sql, server/migrate.sql, server/vps_schema.sql) look like fixtures or reference schemas used during testing and migration.
- The server/test_*.py files indicate the engine is validated at multiple layers: parsing, grouping, endpoint behavior, AI mapping, and migration flow.
- server/cli_map.py and server/debug_regex.py suggest there are command-line and debugging utilities for schema analysis outside the main UI flow.
11. External services that materially affect the design
- The system is built around migration from MySQL source to PostgreSQL target.
- AI assistance is provided through Gemini and possibly Groq for schema interpretation.
- The runtime stack therefore spans browser UI, Node/Next.js API routes, Python migration services, database connectivity, and AI model calls.
12. End-to-end data flow
- A user loads or connects a source schema in the frontend.
- The frontend sends the schema/request through its API routes to the Python engine.
- The engine extracts and parses schemas, computes mappings, optionally consults AI, persists the mapping, and then migrates data in dependency-safe order with ID remapping and conflict handling.
- Results and mapping state are then surfaced back in the dashboard for review and further edits.
Main entry points: complete_migration.py Orchestrator · page.js Workspace · hybrid_mapper.py Hybrid Mapper · main.py Backend Entry
Components 20
Frontend
Next.js operator console and local workspace state
- Workspace · Main migration workspace, schema import, and mapping review entrypoint
frontend/app/page.js - Table Card · Table-level mapping review and editing
frontend/app/TableMappingCard.js - Mapping Detail · Column-level mapping detail editor
frontend/app/TableMappingDetail.js - Engine API · Browser-to-engine command boundary
frontend/app/api/engine/[action]/route.js - Mappings API · Loads and saves mapping state for the workspace
frontend/app/api/mappings/route.js - Projects API · Workspace and project management boundary
frontend/app/api/projects/route.js
Python Engine
Schema analysis, mapping, and migration execution
- Backend Entry · Central backend entrypoint for commands and API execution
server/main.py - Orchestrator · End-to-end migration coordinator
server/complete_migration.py - MySQL Extractor · Live source schema extraction from MySQL
server/mysql_extractor.py - Schema Parser · Parses and normalizes incoming table and column data
server/data_parser.py - Transformers · Prepares rows and schema structures for migration
server/data_transformers.py - Hybrid Mapper · Primary heuristic plus AI-assisted schema matching
server/hybrid_mapper.py - Simple Mapper · Deterministic alternate mapping path
server/simple_mapper.py - AI Mapper · AI-assisted schema correspondence layer
server/ai_schema_mapper.py - ID Remapper · Primary and foreign key remapping for relational integrity
server/id_remapper.py - VPS Connect · Remote target database connection and retry boundary
server/connect_vps.py - CLI Tools · Command-line schema analysis and mapping helpers
server/cli_map.py
Artifacts
Persistent mapping and schema files used across runs
- Local Store · Client-side persisted migration workspace state
frontend/.local-data/migration-store.json - Mapping JSON · Persisted mapping state shared across planning and execution
server/mapping.json - Versioned Maps · Versioned mapping snapshots
server/mappings/v1.0.0.json
Connections 22
- Workspace → Table Card: renders · Dashboard drives table-level review
- Workspace → Mapping Detail: opens · Dashboard drills into column-level mappings
- Workspace → Engine API: requests · Sends schema analysis and migration actions
- Workspace → Mappings API: loads/saves · Syncs mapping state with persistence
- Workspace → Projects API: manages · Handles workspace/project state
- Workspace → Local Store: persists · Local workspace persistence
- Mappings API → Backend Entry: forwards · API boundary into Python backend
- Engine API → Backend Entry: invokes · Executes backend actions by route parameter
- Projects API → Backend Entry: coordinates · Project-level backend operations
- Backend Entry → Orchestrator: orchestrates · Primary backend entry delegates full migrations
- Orchestrator → MySQL Extractor: extracts · Reads source schema from MySQL
- Orchestrator → Schema Parser: parses · Normalizes schema and input payloads
- Orchestrator → Transformers: prepares · Shapes data for execution
- Orchestrator → Hybrid Mapper: maps · Produces table and column correspondences
- Hybrid Mapper → Simple Mapper: falls back · Uses deterministic mapping when needed
- Hybrid Mapper → AI Mapper: assists · Consults model-backed mapping logic
- Orchestrator → ID Remapper: remaps · Preserves relational integrity during move
- Orchestrator → VPS Connect: executes · Targets remote PostgreSQL infrastructure
- Orchestrator → Mapping JSON: writes · Persists migration plan and resolved mappings
- Mapping JSON → Versioned Maps: versions · Stores snapshot variants of mapping state
- CLI Tools → Schema Parser: uses · CLI helper for schema inspection
- CLI Tools → Hybrid Mapper: uses · CLI helper for mapping experiments