# databricks/dbt-databricks architecture

> Architecture diagram of the GitHub repository databricks/dbt-databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databricks/dbt-databricks
- Repository: https://github.com/databricks/dbt-databricks

## Overview

##### Project Type & Purpose
- Type: Open-source adapter/plugin for dbt (data build tool)
- Purpose: Translate dbt’s generic operations (compile, run, test) into Databricks-specific API calls and SQL, supporting Unity Catalog, Delta Lake, Python models, and Databricks features without requiring an ODBC driver.

##### High-Level Architectural Components
a. dbt Core
– Orchestrator: drives compilation, materialization, testing
– Plugin interface: loads the adapter via Python entry points
b. dbt-Databricks Adapter
– Connection Manager: reads profiles, manages credentials, HTTP path, token
– API Client Layer: wraps Databricks REST APIs (clusters, jobs, pipelines, tables)
– Compiler Extension (impl.py, parse_model.py): generates Databricks-flavored SQL, handles incremental/materializations
– Event & Logging Subsystem: emits and logs connection, credential, and pipeline events
– Python Models Support: marshals pandas or pyspark DataFrames, manages compute overrides
– Macro Library (dbt/include/databricks/macros): SQL templates for catalogs, relations, materializations, utilities
c. Databricks Platform
– SQL Warehouses & All-Purpose Clusters: execute SQL or Python models
– Unity Catalog: three-level namespace enforcement and permissions
– Delta Lake: open table format, ACID operations, photon acceleration
d. Testing & CI
– Unit Tests: validate API client behavior, macro output, relation configs
– Functional Tests: spin up ephemeral adapters against a Spark engine or fake Databricks services
– GitHub Actions: linting, unit, integration, coverage checks
e. External Resources
– S3/Data Lakes, Delta tables, Databricks REST endpoint

##### Relationships & Data Flows
- dbt Core → (plugin entry point) → Adapter
- Adapter
→ reads profile → Connection Manager → HTTP token + endpoint
→ compiles model → Compiler Extension + Macro Library → SQL statements or Python model code
→ API Client Layer → REST calls to Databricks (submit SQL, run jobs, fetch metadata)
→ receives results → passes back to dbt Core → logs events
- Databricks Platform executes code → returns success/failure & metadata

##### Architectural Patterns & Principles
– Plugin pattern: adapter implements dbt’s adapter interface
– Layered design: distinct layers for connection, compilation, API, macros, events
– Configuration-as-code: YAML profiles, dbt_project.yml, macro configs
– Extensibility: every materialization and catalog operation lives in macros, overrideable by users
– Test-driven: extensive unit and functional test suites ensure correctness

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% dbt Core
    DBT_Core["dbt Core"]:::core
    DBT_Handle["Plugin Entry Point"]:::core
    DBT_Core -->|"entry point"| DBT_Handle

    %% Adapter Layer
    subgraph "Adapter" 
        direction TB
        ConnManager["Connection Manager"]:::adapter
        APIClient["API Client Layer"]:::adapter
        Compiler["Compiler Extension"]:::adapter
        Parser["Parser Parser"]:::adapter
        Macros["Macro Library"]:::adapter
        EventLogger["Event & Logging Subsystem"]:::adapter
        PythonModels["Python Model Handler"]:::adapter
        Utils["Constants & Utilities"]:::adapter
        Behaviors["Adapter Behaviors"]:::adapter
        Catalogs["Catalog Interfaces"]:::adapter
        RelConfigs["Relation Configurations"]:::adapter
    end
    DBT_Handle -->|"load adapter"| ConnManager
    ConnManager -->|"reads profiles"| Utils
    ConnManager -->|"manages credentials"| ConnManager
    ConnManager -->|"provides token"| APIClient
    Compiler -->|"uses macros"| Macros
    Parser -->|"parses models"| Compiler
    APIClient -->|"submits SQL & jobs"| Databricks_Cloud

    %% Databricks Platform
    subgraph "Databricks Platform"
        direction TB
        SQLWarehouses["SQL Warehouses & Clusters"]:::service
        UnityCatalog["Unity Catalog"]:::dataStore
        DeltaLake["Delta Lake"]:::dataStore
    end
    Databricks_Cloud["Databricks API"]:::service
    APIClient --> Databricks_Cloud
    Databricks_Cloud -->|"returns metadata"| APIClient
    Databricks_Cloud --> SQLWarehouses
    SQLWarehouses -->|"executes code"| Databricks_Cloud

    %% Return to dbt Core
    APIClient -->|"results"| DBT_Core

    %% CI/Test Suite
    subgraph "CI / Test Suite" 
        direction TB
        GitHubActions["GitHub Actions"]:::ci
        UnitTests["Unit Tests"]:::ci
        FuncTests["Functional Tests"]:::ci
    end
    GitHubActions --> UnitTests
    GitHubActions --> FuncTests
    UnitTests -->|"test adapter/macros"| ConnManager
    FuncTests -->|"simulate Databricks"| Databricks_Cloud

    %% Relationships from dbt Core to Tests
    DBT_Core -->|"run tests"| GitHubActions

    %% Click Events
    click DBT_Handle "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/handle.py"
    click ConnManager "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/connections.py"
    click APIClient "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/api_client.py"
    click Compiler "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/impl.py"
    click Parser "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/parse_model.py"
    click EventLogger "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/logging.py"
    click PythonModels "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/python_models/python_config.py"
    click Macros "https://github.com/databricks/dbt-databricks/tree/main/dbt/include/databricks/macros/"
    click RelConfigs "https://github.com/databricks/dbt-databricks/tree/main/dbt/adapters/databricks/relation_configs/"
    click Utils "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/constants.py"
    click Behaviors "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/behaviors/columns.py"
    click Catalogs "https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/catalogs/_unity.py"
    click UnitTests "https://github.com/databricks/dbt-databricks/tree/main/tests/unit/"
    click FuncTests "https://github.com/databricks/dbt-databricks/tree/main/tests/functional/"
    click GitHubActions "https://github.com/databricks/dbt-databricks/blob/main/.github/workflows/ci-pr-linting.yml"

    %% Styles
    classDef core fill:#ADE8F4,stroke:#03045E,color:#000
    classDef adapter fill:#CAF0F8,stroke:#0077B6,color:#000
    classDef service fill:#D8F3DC,stroke:#2D6A4F,color:#000
    classDef ci fill:#FFE5D9,stroke:#D00000,color:#000
    classDef dataStore fill:#F0EFEB,stroke:#444,color:#000
```
