# duckdb/dbt-duckdb architecture

> Architecture diagram of the GitHub repository duckdb/dbt-duckdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/duckdb/dbt-duckdb
- Repository: https://github.com/duckdb/dbt-duckdb

## Overview

##### Main Components
- dbt-duckdb Adapter: This is the core component that connects dbt’s orchestration with the DuckDB database engine. It sits between the dbt core framework and the underlying DuckDB connection.
- DuckDB Engine: Although DuckDB is an embedded database, represent it as an external “database engine” component that the adapter manages. Note that it supports in-memory and file-based databases.
- Profiles & Configuration: Include a component showing how the adapter is configured via dbt profiles (with settings such as “path”, “extensions”, “credentials”, etc.). This is where users specify things like local databases, MotherDuck connection strings, attachment options, external file locations, and fsspec settings.
- Plugins System: Illustrate the extensible plugin framework. This includes built-in plugins (Excel, gsheet, sqlalchemy, delta, etc.) as well as support for user-defined plugins. The system uses a Plugin class (and its initialization, connection configuration, load/store methods) to extend or customize the adapter’s behavior.
- Macros and SQL Materializations: Represent the macros and materializations found in dbt/include/duckdb. These components define how to perform external table materializations (tables, table functions, etc.) and how to handle sources.
- Python Models: Since dbt-duckdb includes support for in-process Python models (batch processing, conversion between Python objects and DuckDB relations), include a module that represents how Python models interact with the adapter and the DuckDB connection.
- Testing and CI/CD: Show that there are dedicated testing directories (unit, functional) and CI/CD configuration (GitHub workflows, devcontainer configuration, pre-commit hooks). These nodes are essential for continuous integration and quality assurance, though they are not “runtime” components, they are part of the overall engineering process.

##### Out Interactions and Relationships
- Adapter → Macro Components: Indicate that the adapter uses SQL macros (from the include directory) to define materializations, external table operations, and table functions. Show the flow of model definitions (SQL/Python) through the adapter.

##### Architectural Patterns and Design Principles
- Environment Differentiation: Consider representing how the same adapter is deployed in various environments (local development using devcontainer, CI/CD pipelines) and how Docker (via .devcontainer and Dockerfile) is used to encapsulate the development environment.

##### Relevant Technologies and Frameworks
- Python: Basis for the adapter’s logic and the testing framework.
- DuckDB: The embedded OLAP database engine.
- dbt Core: The transformation orchestration framework that consumes this adapter.
- CI/CD Tools: GitHub Actions (workflows), pre-commit hooks, and Docker (devcontainer) for development consistency.
- fsspec and External File Processing: Represent components related to external connectors and file system integrations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External and Core Components
    Core["dbt Core"]:::core
    Adapter["dbt-duckdb Adapter"]:::adapter
    DuckDB["DuckDB Engine"]:::database

    %% Configuration/Profiles
    subgraph "Profiles & Configuration"
        Sample["Sample Profile"]:::config
        Creds["Credentials"]:::config
        Secrets["Secrets"]:::config
    end

    %% Plugin System
    Plugin["Plugin System"]:::plugin

    %% Macros and Materializations
    subgraph "SQL Macros & Materializations"
        SQLMacros["SQL Macros"]:::macro
        Materializations["Materializations"]:::macro
    end

    %% Python Models Support
    PyModels["Python Models Support"]:::python

    %% Development and CI/CD Environment
    subgraph "Testing & CI/CD"
        Tests["Test Suite"]:::testing
        Workflows["GitHub Workflows"]:::testing
        DevContainer["Dev Container"]:::testing
        PreCommit["Pre-commit Hooks"]:::testing
    end

    %% Relationships
    Core -->|"triggers"| Adapter
    Sample -->|"configures"| Adapter
    Creds -->|"configures"| Adapter
    Secrets -->|"configures"| Adapter
    Adapter -->|"connects to"| DuckDB
    Adapter -->|"loads"| Plugin
    SQLMacros -->|"used by"| Adapter
    Materializations -->|"used by"| Adapter
    Adapter -->|"executes"| PyModels

    %% CI/CD components testing the Adapter
    Tests -->|"validate"| Adapter
    Workflows -->|"validate"| Adapter
    DevContainer -->|"supports"| Adapter
    PreCommit -->|"enforces"| Adapter

    %% Click events for mapped components
    click Adapter "https://github.com/duckdb/dbt-duckdb/tree/master/dbt/adapters/duckdb"
    click DuckDB "https://github.com/duckdb/dbt-duckdb/blob/master/dbt/adapters/duckdb/connections.py"
    click Sample "https://github.com/duckdb/dbt-duckdb/blob/master/dbt/include/duckdb/sample_profiles.yml"
    click Creds "https://github.com/duckdb/dbt-duckdb/blob/master/dbt/adapters/duckdb/credentials.py"
    click Secrets "https://github.com/duckdb/dbt-duckdb/blob/master/dbt/adapters/duckdb/secrets.py"
    click Plugin "https://github.com/duckdb/dbt-duckdb/tree/master/dbt/adapters/duckdb/plugins"
    click SQLMacros "https://github.com/duckdb/dbt-duckdb/tree/master/dbt/include/duckdb/macros"
    click Materializations "https://github.com/duckdb/dbt-duckdb/tree/master/dbt/include/duckdb/macros/materializations"
    click PyModels "https://github.com/duckdb/dbt-duckdb/blob/master/tests/functional/adapter/test_python_model.py"
    click Tests "https://github.com/duckdb/dbt-duckdb/tree/master/tests"
    click Workflows "https://github.com/duckdb/dbt-duckdb/tree/master/.github/workflows"
    click DevContainer "https://github.com/duckdb/dbt-duckdb/blob/master/.devcontainer"
    click PreCommit "https://github.com/duckdb/dbt-duckdb/blob/master/.pre-commit-config.yaml"

    %% Styles
    classDef core fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef adapter fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef database fill:#F08080,stroke:#333,stroke-width:2px;
    classDef config fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef plugin fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef macro fill:#FFA500,stroke:#333,stroke-width:2px;
    classDef python fill:#C0C0C0,stroke:#333,stroke-width:2px;
    classDef testing fill:#D8BFD8,stroke:#333,stroke-width:2px;
```
