# databrickslabs/migrate architecture

> Architecture diagram of the GitHub repository databrickslabs/migrate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databrickslabs/migrate
- Repository: https://github.com/databrickslabs/migrate

## Overview

##### Project Type and Purpose
- This is an open-source, full-workspace migration tool written in Python.
- Purpose: Export and import Databricks workspace artifacts (users/groups, clusters, notebooks, jobs, metastore, ACLs, MLflow, etc.) across workspaces, regions, or cloud providers.

##### File Structure
Top-level modules & directories:
- export_db.py & import_db.py – CLI entry points for one-off export/import steps.
- migration_pipeline.py (in root and pipeline/) – Orchestrator for the end-to-end export/import pipeline with checkpointing and parallelism.
- dbclient/ – Abstractions for all Databricks REST API calls (ClustersClient, JobsClient, HiveClient, MLFlowClient, etc.).
- tasks/ & pipeline/ – Task definitions and pipeline engine invoking tasks in sequence or parallel.
- data/ – JSON configuration templates and Spark notebooks used for metastore & table-ACL exports.
- validate/ – Post-import validation logic (json diff).
- jumpbox/ & docker/ – Scripts and Dockerfile to deploy a controlled execution environment.
- threading_utils.py & thread_safe_writer.py – Concurrency support.
- checkpoint_service.py – Checkpoint management for pipeline restarts.

##### Extract Key Technologies and Patterns from README
- Python 3.6+ with multiprocessing/threading.
- Databricks CLI for authentication & REST API.
- Spark Execution Context for metastore & table ACL notebooks.
- Pipeline pattern with checkpointing, retry/backoff, parallel task execution.
- SCIM API, Clusters API, Jobs API, Workspace API.

##### Main Components
A. CLI Layer
– export_db.py
– import_db.py
– migration_pipeline.py
B. Orchestration Layer
– Pipeline Engine (pipeline/pipeline.py, task.py)
– Checkpoint Service (checkpoint_service.py)
– Task Scheduler & Worker Threads (threading_utils.py, thread_safe_writer.py)
C. Task Layer
– Task modules (tasks/tasks.py, pipeline tasks)
– Each task corresponds to one migration component: users, clusters, notebooks, jobs, metastore, ACLs, secrets, MLflow.
D. API Client Layer
– dbclient package (ClustersClient, JobsClient, HiveClient, etc.)
– Parser & common utilities (dbclient/parser.py, common/)
E. Data & Notebook Artifacts
– data/ JSON definitions (cluster profiles, default jobs…)
– data/notebooks Spark notebooks for metastore & ACL export/import
– logs/ export-dir where tasks write JSON/log files
F. Validation Layer
– validate/json_diff.py
– validate_pipeline.sh
G. Infrastructure & Environment
– Docker environment (docker/Dockerfile, docker/databricks/.databrickscfg)
– Jumpbox scripts for secure SSH bastion usage (jumpbox/)

##### Relationships & Data Flow
1. User invokes CLI →
2. CLI parses arguments & instantiates Pipeline Engine or individual TaskRunner →
3. Pipeline Engine reads checkpoint info →
4. Scheduler dispatches Task objects (e.g., export users, export clusters) →
5. Each Task calls the appropriate dbclient.*Client to make REST calls against source or destination Databricks workspace →
6. Responses are written to the export directory (JSON, logs, notebook files) using thread_safe_writer →
7. After export, optional transformation step (export_db.py with account-id update) →
8. Import pipeline tasks consume JSON/log files, call REST API on destination workspace →
9. Validation layer reads both source & destination logs via validate/json_diff →
10. Report generation.

##### Architectural Patterns & Principles
- Pipeline pattern with pluggable, idempotent tasks.
- Client abstraction isolating API details (single responsibility).
- Checkpointing for fault-tolerance and restartability.
- Thread-safe I/O and retryable HTTP calls for resilience.
- Separation of concerns: orchestration vs. tasks vs. API clients.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        ExportCLI["export_db.py"]:::cli
        ImportCLI["import_db.py"]:::cli
        MigrateCLI["migration_pipeline.py"]:::cli
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        PipelineEngine["Pipeline Engine"]:::orchestration
        TaskDef["Task Definition"]:::orchestration
        CheckpointService["Checkpoint Service"]:::orchestration
        ThreadUtils["Threading Utils"]:::orchestration
        TSWriter["Thread Safe Writer"]:::orchestration
    end

    %% Task Layer
    subgraph "Task Layer"
        TasksImpl["Task Implementations"]:::tasks
    end

    %% API Client Layer
    subgraph "API Client Layer"
        dbclientEntry["dbclient/dbclient.py"]:::client
        ClustersClient["ClustersClient"]:::client
        DbfsClient["DbfsClient"]:::client
        HiveClient["HiveClient"]:::client
        JobsClient["JobsClient"]:::client
        LibraryClient["LibraryClient"]:::client
        MLFlowClient["MLFlowClient"]:::client
        ScimClient["ScimClient"]:::client
        SecretsClient["SecretsClient"]:::client
        TableACLsClient["TableACLsClient"]:::client
        WorkspaceClient["WorkspaceClient"]:::client
        Parser["parser.py"]:::client
        ViewSort["ViewSort.py"]:::client
        WorkspaceDiff["WorkspaceDiff.py"]:::client
    end

    %% Data & Notebook Artifacts
    subgraph "Data & Notebook Artifacts"
        JSONConfig["JSON Config Templates"]:::data
        Notebooks["Spark Notebooks"]:::data
        ExportDir["Export Directory (logs/JSON)"]:::storage
    end

    %% Validation Layer
    subgraph "Validation Layer"
        JSONDiff["JSON Diff Logic"]:::validation
        ValidateScript["validate_pipeline.sh"]:::validation
    end

    %% Infrastructure & Environment
    subgraph "Infrastructure & Environment"
        DockerEnv["Docker Environment"]:::infra
        Jumpbox["Jumpbox Scripts"]:::infra
    end

    %% External Systems
    SourceWS["Source Databricks Workspace"]:::external
    DestWS["Destination Databricks Workspace"]:::external
    DatabricksCLI["Databricks CLI"]:::external
    SparkContext["Spark Execution Context"]:::external

    %% Connections
    ExportCLI -->|invokes| PipelineEngine
    ImportCLI -->|invokes| PipelineEngine
    MigrateCLI -->|invokes| PipelineEngine
    DockerEnv -->|hosts| ExportCLI
    DockerEnv -->|hosts| ImportCLI
    DockerEnv -->|hosts| MigrateCLI

    PipelineEngine -->|reads/writes checkpoints| CheckpointService
    PipelineEngine -->|schedules tasks| TaskDef
    PipelineEngine -->|uses threads| ThreadUtils
    PipelineEngine -->|writes output| TSWriter

    TaskDef -->|creates| TasksImpl
    TasksImpl -->|calls| dbclientEntry
    dbclientEntry --> ClustersClient
    dbclientEntry --> DbfsClient
    dbclientEntry --> HiveClient
    dbclientEntry --> JobsClient
    dbclientEntry --> LibraryClient
    dbclientEntry --> MLFlowClient
    dbclientEntry --> ScimClient
    dbclientEntry --> SecretsClient
    dbclientEntry --> TableACLsClient
    dbclientEntry --> WorkspaceClient
    dbclientEntry --> Parser
    dbclientEntry --> ViewSort
    dbclientEntry --> WorkspaceDiff

    ClustersClient -->|HTTP REST calls| SourceWS
    DbfsClient -->|HTTP REST calls| SourceWS
    HiveClient -->|HTTP REST calls| SourceWS
    JobsClient -->|HTTP REST calls| SourceWS
    LibraryClient -->|HTTP REST calls| SourceWS
    MLFlowClient -->|HTTP REST calls| SourceWS
    ScimClient -->|HTTP REST calls| SourceWS
    SecretsClient -->|HTTP REST calls| SourceWS
    TableACLsClient -->|HTTP REST calls| SourceWS
    WorkspaceClient -->|HTTP REST calls| SourceWS

    TasksImpl -->|writes artifacts| ExportDir
    Notebooks -->|executed on| SparkContext
    SparkContext -->|runs notebooks against| SourceWS

    ImportCLI -->|reads artifacts| ExportDir
    ImportCLI -->|invokes| PipelineEngine

    dbclientEntry --> ClustersClient
    %% import clients to dest
    ClustersClient -->|HTTP REST calls| DestWS
    DbfsClient -->|HTTP REST calls| DestWS
    HiveClient -->|HTTP REST calls| DestWS
    JobsClient -->|HTTP REST calls| DestWS
    LibraryClient -->|HTTP REST calls| DestWS
    MLFlowClient -->|HTTP REST calls| DestWS
    ScimClient -->|HTTP REST calls| DestWS
    SecretsClient -->|HTTP REST calls| DestWS
    TableACLsClient -->|HTTP REST calls| DestWS
    WorkspaceClient -->|HTTP REST calls| DestWS

    %% Validation
    JSONDiff -->|reads logs| ExportDir
    JSONDiff -->|reads dest logs| ExportDir
    ValidateScript -->|runs| JSONDiff

    %% Click Events
    click ExportCLI "https://github.com/databrickslabs/migrate/blob/master/export_db.py"
    click ImportCLI "https://github.com/databrickslabs/migrate/blob/master/import_db.py"
    click MigrateCLI "https://github.com/databrickslabs/migrate/blob/master/migration_pipeline.py"
    click PipelineEngine "https://github.com/databrickslabs/migrate/blob/master/pipeline/pipeline.py"
    click TaskDef "https://github.com/databrickslabs/migrate/blob/master/pipeline/task.py"
    click CheckpointService "https://github.com/databrickslabs/migrate/blob/master/checkpoint_service.py"
    click ThreadUtils "https://github.com/databrickslabs/migrate/blob/master/threading_utils.py"
    click TSWriter "https://github.com/databrickslabs/migrate/blob/master/thread_safe_writer.py"
    click TasksImpl "https://github.com/databrickslabs/migrate/blob/master/tasks/tasks.py"
    click dbclientEntry "https://github.com/databrickslabs/migrate/blob/master/dbclient/dbclient.py"
    click ClustersClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/ClustersClient.py"
    click DbfsClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/DbfsClient.py"
    click HiveClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/HiveClient.py"
    click JobsClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/JobsClient.py"
    click LibraryClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/LibraryClient.py"
    click MLFlowClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/MLFlowClient.py"
    click ScimClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/ScimClient.py"
    click SecretsClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/SecretsClient.py"
    click TableACLsClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/TableACLsClient.py"
    click WorkspaceClient "https://github.com/databrickslabs/migrate/blob/master/dbclient/WorkspaceClient.py"
    click Parser "https://github.com/databrickslabs/migrate/blob/master/dbclient/parser.py"
    click ViewSort "https://github.com/databrickslabs/migrate/blob/master/dbclient/common/ViewSort.py"
    click WorkspaceDiff "https://github.com/databrickslabs/migrate/blob/master/dbclient/common/WorkspaceDiff.py"
    click JSONConfig "https://github.com/databrickslabs/migrate/tree/master/data"
    click Notebooks "https://github.com/databrickslabs/migrate/tree/master/data/notebooks"
    click JSONDiff "https://github.com/databrickslabs/migrate/blob/master/validate/json_diff.py"
    click ValidateScript "https://github.com/databrickslabs/migrate/blob/master/validate_pipeline.sh"
    click DockerEnv "https://github.com/databrickslabs/migrate/tree/master/docker"
    click Jumpbox "https://github.com/databrickslabs/migrate/tree/master/jumpbox"

    %% Styles
    classDef cli fill:#d8b4fe,stroke:#8b5cf6,color:#000
    classDef orchestration fill:#bfdbfe,stroke:#3b82f6,color:#000
    classDef tasks fill:#bae6fd,stroke:#0284c7,color:#000
    classDef client fill:#dcfce7,stroke:#22c55e,color:#000
    classDef data fill:#fed7aa,stroke:#fb923c,color:#000
    classDef storage fill:#f5f3ff,stroke:#7c3aed,color:#000
    classDef validation fill:#ffe4e6,stroke:#ec4899,color:#000
    classDef infra fill:#e5e7eb,stroke:#6b7280,color:#000
    classDef external fill:#f0fdf4,stroke:#10b981,color:#000
```
