# databricks/databricks-sdk-py architecture

> Architecture diagram of the GitHub repository databricks/databricks-sdk-py, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databricks/databricks-sdk-py
- Repository: https://github.com/databricks/databricks-sdk-py

## Overview

##### Project type and purpose
- This is an open-source Python SDK (library) for Databricks.
- It provides a high-level client (`WorkspaceClient`, `AccountClient`) that wraps all public Databricks REST APIs, handles authentication flows, retries, pagination, and exposes every service (clusters, jobs, SQL, etc.) as strongly typed Python methods.

##### File-structure analysis
- Top-level:
- `databricks/sdk/` – core runtime code, subdivided into authentication, HTTP client, service definitions, mixins, errors, logging, runtime stubs.
- `examples/` – sample scripts showing how to use the SDK (OAuth flow, long-running ops, etc.).
- `docs/` – Sphinx-based docs organized by service and topic.
- `tests/` – unit and integration tests, including a fixture HTTP server.
- Config and packaging: `pyproject.toml`, `setup.cfg`, `Makefile`, GitHub workflows, codegen metadata.

##### Key insights from README
- Main clients: `WorkspaceClient` (workspace-level) and `AccountClient` (account-level).
- Authentication strategies: native (PAT), OIDC, Azure AD, GCP credentials, External-Browser OAuth, PKCE for web apps.
- Retry logic, pagination, long-running-ops abstraction (`Waiter` objects).
- `dbutils` shim under `databricks/sdk/runtime`.

A. SDK Layer (your code):
1. User code / applications (examples, customer scripts)
2. Public API Facade: `WorkspaceClient`, `AccountClient`
3. Service Clients: modules under `databricks/sdk/service/*` (clusters, jobs, sql, iam, etc.)
4. Mixins: high-level convenience methods under `databricks/sdk/mixins/*`
5. Core HTTP client: `databricks/sdk/_base_client.py`, handles serialization, retries, timeouts, user-agent, logging
6. Authentication providers:
– Native token and OIDC (`credentials_provider.py`, `.oauth`, `.oidc`)
– Azure (`oauth.py`, Azure mixin, environment variables)
– GCP (`google_credentials`, service account)
– External-Browser / PKCE (`oauth_client.py`)
7. Errors and Exceptions: `databricks/sdk/errors/*` (consistent error model)
8. Runtime stubs: `databricks/sdk/runtime/dbutils_stub.py` and `databricks/sdk/runtime` for notebook integration
B. External Systems:
1. Databricks Account Control Plane REST API
2. Databricks Workspace Data Plane REST API
3. OAuth Authorization Server (for PKCE flow)
4. Azure CLI / GCP DAC (for native cloud auth flows)

##### Relationships and interactions
- User code → calls `WorkspaceClient` / `AccountClient` → routes to appropriate Service Client.
- Service Client → uses HTTP Client → signs requests with chosen Authentication Provider → issues REST calls to Databricks endpoints.
- HTTP Client → implements retry logic, pagination, logging, user-agent enrichment.
- Long-running ops (clusters, jobs) return `Waiter` objects that poll the REST API.
- Runtime stubs (`dbutils`) call back into Service Clients to implement file/secret utilities.

##### Architectural patterns and principles
- Facade pattern: `WorkspaceClient` / `AccountClient` present a unified entry point.
- Adapter pattern: each service module adapts raw REST endpoints into Python classes.
- Strategy pattern: pluggable authentication strategies.
- Template Method: HTTP client defines request/response flow, error handling, retry.
- Separation of concerns: transport, auth, service definitions, user code/examples, docs, tests are strictly decoupled.

##### Technologies and frameworks
- Python 3.7–3.11
- HTTP: `requests` or internal transport
- Sphinx docs, pytest for tests, GitHub Actions for CI, codegen from OpenAPI specs.

##### Final advice
- Be exhaustive: list every internal module group.
- External dependencies (Databricks endpoints, OAuth server, cloud CLIs).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers Top-down
    subgraph "Application Layer"
        direction TB
        User["User Code / Applications"]:::ext
    end

    subgraph "Public API Facade"
        direction TB
        WC["WorkspaceClient"]:::publicapi
        AC["AccountClient"]:::publicapi
    end

    subgraph "Service Clients"
        direction TB
        SC["Service Clients\n(e.g., clusters, jobs, sql, iam, pipelines)"]:::sdk
    end

    subgraph "Mixins"
        direction TB
        MX["Mixins\n(compute, files, jobs, workspace, open_ai_client, sharing)"]:::sdk
    end

    subgraph "Core HTTP & Support"
        direction TB
        HTTP["HTTP Client\n(databricks/sdk/_base_client.py)"]:::sdk
        AuthProv["Auth Providers"]:::sdk
        Err["Errors & Exceptions\n(databricks/sdk/errors/)"]:::sdk
        Retry["Retry Logic\n(databricks/sdk/retries.py)"]:::sdk
        Logger["Round-Trip Logger\n(databricks/sdk/logger/round_trip_logger.py)"]:::sdk
        Config["Configuration & Env\n(databricks/sdk/config.py,\n/environments.py)"]:::sdk
        UA["User-Agent Composer\n(databricks/sdk/useragent.py)"]:::sdk
        Runtime["Runtime Stubs\n(databricks/sdk/runtime/dbutils_stub.py)"]:::sdk
    end

    subgraph "External Systems"
        direction TB
        DBAcct["Databricks Account REST API"]:::ext
        DBWS["Databricks Workspace REST API"]:::ext
        OAuthSrv["OAuth Authorization Server"]:::ext
        CloudCLI["Azure CLI / GCP CLI"]:::ext
    end

    %% Connections
    User -->|uses| WC
    User -->|uses| AC

    WC -->|calls| SC
    AC -->|calls| SC

    SC -->|issues| HTTP

    HTTP -->|signs via| AuthProv
    HTTP -->|handles errors| Err
    HTTP -->|applies retry| Retry
    HTTP -->|logs via| Logger
    HTTP -->|reads config| Config
    HTTP -->|adds UA header| UA

    AuthProv -->|may call| OAuthSrv
    AuthProv -->|may call| CloudCLI

    HTTP -->|sends REST| DBAcct
    HTTP -->|sends REST| DBWS

    Runtime -->|calls| SC

    %% Click events
    click WC "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/core.py"
    click AC "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/core.py"
    click SC "https://github.com/databricks/databricks-sdk-py/tree/main/databricks/sdk/service/"
    click MX "https://github.com/databricks/databricks-sdk-py/tree/main/databricks/sdk/mixins/"
    click HTTP "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/_base_client.py"
    click Err "https://github.com/databricks/databricks-sdk-py/tree/main/databricks/sdk/errors/"
    click Retry "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/retries.py"
    click Logger "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/logger/round_trip_logger.py"
    click Config "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/config.py"
    click Config "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/environments.py"
    click UA "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/useragent.py"
    click Runtime "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/runtime/dbutils_stub.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/credentials_provider.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/oidc.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/oidc_token_supplier.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/oauth.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/azure.py"
    click AuthProv "https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/environments.py"

    %% Styles
    classDef publicapi fill:#8FBC8F,stroke:#333,stroke-width:1px
    classDef sdk fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef ext fill:#FFA500,stroke:#333,stroke-width:1px
```
