How databricks/dbt-databricks works
Project Type & Purpose
• Type: Open-source adapter/plugin for dbt (data build tool)
• Purpose: Translate dbt’s generic operations (compile, run, test) into Databricks-specific API calls and SQL, supporting Unity Catalog, Delta Lake, Python models, and Databricks features without requiring an ODBC driver.
High-Level Architectural Components
a. dbt Core
– Orchestrator: drives compilation, materialization, testing
– Plugin interface: loads the adapter via Python entry points
b. dbt-Databricks Adapter
– Connection Manager: reads profiles, manages credentials, HTTP path, token
– API Client Layer: wraps Databricks REST APIs (clusters, jobs, pipelines, tables)
– Compiler Extension (impl.py, parse_model.py): generates Databricks-flavored SQL, handles incremental/materializations
– Event & Logging Subsystem: emits and logs connection, credential, and pipeline events
– Python Models Support: marshals pandas or pyspark DataFrames, manages compute overrides
– Macro Library (dbt/include/databricks/macros): SQL templates for catalogs, relations, materializations, utilities
c. Databricks Platform
– SQL Warehouses & All-Purpose Clusters: execute SQL or Python models
– Unity Catalog: three-level namespace enforcement and permissions
– Delta Lake: open table format, ACID operations, photon acceleration
d. Testing & CI
– Unit Tests: validate API client behavior, macro output, relation configs
– Functional Tests: spin up ephemeral adapters against a Spark engine or fake Databricks services
– GitHub Actions: linting, unit, integration, coverage checks
e. External Resources
– S3/Data Lakes, Delta tables, Databricks REST endpoint
Relationships & Data Flows
• dbt Core → (plugin entry point) → Adapter
• Adapter
→ reads profile → Connection Manager → HTTP token + endpoint
→ compiles model → Compiler Extension + Macro Library → SQL statements or Python model code
→ API Client Layer → REST calls to Databricks (submit SQL, run jobs, fetch metadata)
→ receives results → passes back to dbt Core → logs events
• Databricks Platform executes code → returns success/failure & metadata
Architectural Patterns & Principles
– Plugin pattern: adapter implements dbt’s adapter interface
– Layered design: distinct layers for connection, compilation, API, macros, events
– Configuration-as-code: YAML profiles, dbt_project.yml, macro configs
– Extensibility: every materialization and catalog operation lives in macros, overrideable by users
– Test-driven: extensive unit and functional test suites ensure correctness