# databricks/cli architecture

> Architecture diagram of the GitHub repository databricks/cli, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databricks/cli
- Repository: https://github.com/databricks/cli

## Overview

Databricks CLI exposes command-line operations against Databricks services and supports defining, generating, deploying, and running Databricks Asset Bundles. The clearest sampled workflow is pipeline management: users invoke workspace commands or bundle commands; bundle generation can fetch an existing pipeline and download its libraries and root-path files, while bundle runtime code can start updates and poll for status. Bundle configuration, resource modeling, validation, and Python pipeline models form supporting layers. Evidence is strongest for the shown pipeline calls; other command and deployment relationships are intentionally left unconnected where excerpts do not establish them.

Main entry points: [`python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py`](https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py) (Ingestion definitions), [`python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py`](https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py) (Gateway definition), [`cmd/bundle/generate/pipeline.go`](https://github.com/databricks/cli/blob/main/cmd/bundle/generate/pipeline.go) (Generate pipeline), [`bundle/run/pipeline.go`](https://github.com/databricks/cli/blob/main/bundle/run/pipeline.go) (Pipeline update runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_commands["CLI commands"]
  node_workspace_pipeline_cmd["Workspace pipelines<br/>[pipelines.go]"]
  node_feature_cmd["Feature engineering"]
  node_storage_cmd["Account storage<br/>[storage.go]"]
  node_generate_cmd["Generate pipeline<br/>[pipeline.go]"]
end

subgraph group_bundles["Asset Bundles"]
  node_bundle_engine["Config engine<br/>[engine.go]"]
  node_pipeline_config["Pipeline config<br/>[pipeline.go]"]
  node_pipeline_validation["Development validation"]
  node_pipeline_metadata["Pipeline metadata"]
  node_pipeline_resource["Direct resource<br/>[pipeline.go]"]
  node_pipeline_generator["Config conversion<br/>[pipeline.go]"]
end

subgraph group_runtime["Pipeline runtime"]
  node_bundle_run["Pipeline update runner<br/>[pipeline.go]"]
  node_run_progress["Progress reporting<br/>[pipeline.go]"]
  node_local_python["Local Python setup<br/>[pipeline.go]"]
end

subgraph group_python["Python bundle models"]
  node_core_bundle["Bundle core<br/>[_bundle.py]"]
  node_core_location["Bundle locations<br/>[_location.py]"]
  node_core_transform["Model transforms<br/>[_transform.py]"]
  node_core_json["JSON serialization"]
  node_core_variable["Bundle variables<br/>[_variable.py]"]
  node_pipeline_models["Pipeline model set"]
  node_ingestion_model["Ingestion definitions"]
  node_gateway_model["Gateway definition"]
end

subgraph group_services["Databricks services"]
  node_workspace_api["Workspace API"]
  node_account_api["Account API"]
end

node_user(("CLI user"))

node_user -->|"invokes"| node_workspace_pipeline_cmd
node_user -->|"invokes"| node_feature_cmd
node_user -->|"invokes"| node_storage_cmd
node_user -->|"invokes"| node_generate_cmd
node_generate_cmd -->|"fetches pipeline"| node_workspace_api
node_generate_cmd -->|"converts spec"| node_pipeline_generator
node_bundle_run -->|"starts and polls"| node_workspace_api
node_bundle_run -.->|"reports progress"| node_run_progress
node_pipeline_resource -->|"manages pipeline"| node_workspace_api
node_ingestion_model -->|"transforms from dict"| node_core_transform
node_ingestion_model -->|"serializes to JSON"| node_core_json
node_ingestion_model -->|"supports variables"| node_core_variable
node_gateway_model -->|"transforms from dict"| node_core_transform
node_gateway_model -->|"serializes to JSON"| node_core_json
node_gateway_model -->|"supports variables"| node_core_variable
node_storage_cmd -->|"calls API"| node_account_api
node_workspace_pipeline_cmd -.->|"calls API"| node_workspace_api
node_feature_cmd -.->|"calls API"| node_workspace_api

click node_workspace_pipeline_cmd "https://github.com/databricks/cli/blob/main/cmd/workspace/pipelines/pipelines.go"
click node_feature_cmd "https://github.com/databricks/cli/blob/main/cmd/workspace/feature-engineering/feature-engineering.go"
click node_storage_cmd "https://github.com/databricks/cli/blob/main/cmd/account/storage/storage.go"
click node_generate_cmd "https://github.com/databricks/cli/blob/main/cmd/bundle/generate/pipeline.go"
click node_bundle_engine "https://github.com/databricks/cli/blob/main/bundle/config/engine/engine.go"
click node_pipeline_config "https://github.com/databricks/cli/blob/main/bundle/config/resources/pipeline.go"
click node_pipeline_validation "https://github.com/databricks/cli/blob/main/bundle/config/validate/pipeline_development_deprecated.go"
click node_pipeline_metadata "https://github.com/databricks/cli/blob/main/bundle/deploy/metadata/annotate_pipelines.go"
click node_pipeline_resource "https://github.com/databricks/cli/blob/main/bundle/direct/dresources/pipeline.go"
click node_pipeline_generator "https://github.com/databricks/cli/blob/main/bundle/generate/pipeline.go"
click node_bundle_run "https://github.com/databricks/cli/blob/main/bundle/run/pipeline.go"
click node_run_progress "https://github.com/databricks/cli/blob/main/bundle/run/progress/pipeline.go"
click node_local_python "https://github.com/databricks/cli/blob/main/libs/localenv/pipeline.go"
click node_core_bundle "https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_bundle.py"
click node_core_location "https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_location.py"
click node_core_transform "https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_transform.py"
click node_core_json "https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_transform_to_json.py"
click node_core_variable "https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_variable.py"
click node_pipeline_models "https://github.com/databricks/cli/tree/main/python/databricks/bundles/pipelines/_models"
click node_ingestion_model "https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py"
click node_gateway_model "https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workspace_pipeline_cmd,node_feature_cmd,node_storage_cmd,node_generate_cmd,node_user toneBlue
class node_bundle_engine,node_pipeline_config,node_pipeline_validation,node_pipeline_metadata,node_pipeline_resource,node_pipeline_generator toneAmber
class node_bundle_run,node_run_progress,node_local_python toneMint
class node_core_bundle,node_core_location,node_core_transform,node_core_json,node_core_variable,node_pipeline_models,node_ingestion_model,node_gateway_model toneRose
class node_workspace_api,node_account_api toneIndigo
```

## Components

### CLI commands

- [Workspace pipelines](https://github.com/databricks/cli/blob/main/cmd/workspace/pipelines/pipelines.go): `cmd/workspace/pipelines/pipelines.go`
- [Feature engineering](https://github.com/databricks/cli/blob/main/cmd/workspace/feature-engineering/feature-engineering.go): `cmd/workspace/feature-engineering/feature-engineering.go`
- [Account storage](https://github.com/databricks/cli/blob/main/cmd/account/storage/storage.go): `cmd/account/storage/storage.go`
- [Generate pipeline](https://github.com/databricks/cli/blob/main/cmd/bundle/generate/pipeline.go): `cmd/bundle/generate/pipeline.go`

### Asset Bundles

- [Config engine](https://github.com/databricks/cli/blob/main/bundle/config/engine/engine.go): `bundle/config/engine/engine.go`
- [Pipeline config](https://github.com/databricks/cli/blob/main/bundle/config/resources/pipeline.go): `bundle/config/resources/pipeline.go`
- [Development validation](https://github.com/databricks/cli/blob/main/bundle/config/validate/pipeline_development_deprecated.go): `bundle/config/validate/pipeline_development_deprecated.go`
- [Pipeline metadata](https://github.com/databricks/cli/blob/main/bundle/deploy/metadata/annotate_pipelines.go): `bundle/deploy/metadata/annotate_pipelines.go`
- [Direct resource](https://github.com/databricks/cli/blob/main/bundle/direct/dresources/pipeline.go): `bundle/direct/dresources/pipeline.go`
- [Config conversion](https://github.com/databricks/cli/blob/main/bundle/generate/pipeline.go): `bundle/generate/pipeline.go`

### Pipeline runtime

- [Pipeline update runner](https://github.com/databricks/cli/blob/main/bundle/run/pipeline.go): `bundle/run/pipeline.go`
- [Progress reporting](https://github.com/databricks/cli/blob/main/bundle/run/progress/pipeline.go): `bundle/run/progress/pipeline.go`
- [Local Python setup](https://github.com/databricks/cli/blob/main/libs/localenv/pipeline.go): `libs/localenv/pipeline.go`

### Python bundle models

- [Bundle core](https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_bundle.py): `python/databricks/bundles/core/_bundle.py`
- [Bundle locations](https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_location.py): `python/databricks/bundles/core/_location.py`
- [Model transforms](https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_transform.py): `python/databricks/bundles/core/_transform.py`
- [JSON serialization](https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_transform_to_json.py): `python/databricks/bundles/core/_transform_to_json.py`
- [Bundle variables](https://github.com/databricks/cli/blob/main/python/databricks/bundles/core/_variable.py): `python/databricks/bundles/core/_variable.py`
- [Pipeline model set](https://github.com/databricks/cli/tree/main/python/databricks/bundles/pipelines/_models): `python/databricks/bundles/pipelines/_models`
- [Ingestion definitions](https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py): `python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py`
- [Gateway definition](https://github.com/databricks/cli/blob/main/python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py): `python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py`

### Databricks services

- **Workspace API**
- **Account API**

### Other components

- **CLI user**

## Connections

- CLI user → Workspace pipelines: invokes (evidence: `README.md`)
- CLI user → Feature engineering: invokes (evidence: `README.md`)
- CLI user → Account storage: invokes (evidence: `README.md`)
- CLI user → Generate pipeline: invokes (evidence: `README.md`)
- Generate pipeline → Workspace API: fetches pipeline (evidence: `cmd/bundle/generate/pipeline.go`)
- Generate pipeline → Config conversion: converts spec (evidence: `cmd/bundle/generate/pipeline.go`)
- Pipeline update runner → Workspace API: starts and polls (evidence: `bundle/run/pipeline.go`)
- Pipeline update runner → Progress reporting: reports progress
- Direct resource → Workspace API: manages pipeline (evidence: `bundle/direct/dresources/pipeline.go`)
- Ingestion definitions → Model transforms: transforms from dict (evidence: `python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py`)
- Ingestion definitions → JSON serialization: serializes to JSON (evidence: `python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py`)
- Ingestion definitions → Bundle variables: supports variables (evidence: `python/databricks/bundles/pipelines/_models/ingestion_pipeline_definition.py`)
- Gateway definition → Model transforms: transforms from dict (evidence: `python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py`)
- Gateway definition → JSON serialization: serializes to JSON (evidence: `python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py`)
- Gateway definition → Bundle variables: supports variables (evidence: `python/databricks/bundles/pipelines/_models/ingestion_gateway_pipeline_definition.py`)
- Account storage → Account API: calls API (evidence: `cmd/account/storage/storage.go`)
- Workspace pipelines → Workspace API: calls API
- Feature engineering → Workspace API: calls API
