How databricks/databricks-sdk-py works
Project type and purpose
• This is an open-source Python SDK (library) for Databricks.
• It provides a high-level client (WorkspaceClient, AccountClient) that wraps all public Databricks REST APIs, handles authentication flows, retries, pagination, and exposes every service (clusters, jobs, SQL, etc.) as strongly typed Python methods.
File-structure analysis
• Top-level:
• databricks/sdk/ – core runtime code, subdivided into authentication, HTTP client, service definitions, mixins, errors, logging, runtime stubs.
• examples/ – sample scripts showing how to use the SDK (OAuth flow, long-running ops, etc.).
• docs/ – Sphinx-based docs organized by service and topic.
• tests/ – unit and integration tests, including a fixture HTTP server.
• Config and packaging: pyproject.toml, setup.cfg, Makefile, GitHub workflows, codegen metadata.
Key insights from README
• Main clients: WorkspaceClient (workspace-level) and AccountClient (account-level).
• Authentication strategies: native (PAT), OIDC, Azure AD, GCP credentials, External-Browser OAuth, PKCE for web apps.
• Retry logic, pagination, long-running-ops abstraction (Waiter objects).
• dbutils shim under databricks/sdk/runtime.
A. SDK Layer (your code):
1. User code / applications (examples, customer scripts)
2. Public API Facade: WorkspaceClient, AccountClient
3. Service Clients: modules under databricks/sdk/service/* (clusters, jobs, sql, iam, etc.)
4. Mixins: high-level convenience methods under databricks/sdk/mixins/*
5. Core HTTP client: databricks/sdk/_base_client.py, handles serialization, retries, timeouts, user-agent, logging
6. Authentication providers:
– Native token and OIDC (credentials_provider.py, .oauth, .oidc)
– Azure (oauth.py, Azure mixin, environment variables)
– GCP (google_credentials, service account)
– External-Browser / PKCE (oauth_client.py)
7. Errors and Exceptions: databricks/sdk/errors/* (consistent error model)
8. Runtime stubs: databricks/sdk/runtime/dbutils_stub.py and databricks/sdk/runtime for notebook integration
B. External Systems:
1. Databricks Account Control Plane REST API
2. Databricks Workspace Data Plane REST API
3. OAuth Authorization Server (for PKCE flow)
4. Azure CLI / GCP DAC (for native cloud auth flows)
Relationships and interactions
• User code → calls WorkspaceClient / AccountClient → routes to appropriate Service Client.
• Service Client → uses HTTP Client → signs requests with chosen Authentication Provider → issues REST calls to Databricks endpoints.
• HTTP Client → implements retry logic, pagination, logging, user-agent enrichment.
• Long-running ops (clusters, jobs) return Waiter objects that poll the REST API.
• Runtime stubs (dbutils) call back into Service Clients to implement file/secret utilities.
Architectural patterns and principles
• Facade pattern: WorkspaceClient / AccountClient present a unified entry point.
• Adapter pattern: each service module adapts raw REST endpoints into Python classes.
• Strategy pattern: pluggable authentication strategies.
• Template Method: HTTP client defines request/response flow, error handling, retry.
• Separation of concerns: transport, auth, service definitions, user code/examples, docs, tests are strictly decoupled.
Technologies and frameworks
• Python 3.7–3.11
• HTTP: requests or internal transport
• Sphinx docs, pytest for tests, GitHub Actions for CI, codegen from OpenAPI specs.
Final advice
• Be exhaustive: list every internal module group.
• External dependencies (Databricks endpoints, OAuth server, cloud CLIs).