# dbt-labs/dbt-codegen architecture

> Architecture diagram of the GitHub repository dbt-labs/dbt-codegen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbt-labs/dbt-codegen
- Repository: https://github.com/dbt-labs/dbt-codegen

## Overview

##### Main Components
- Developer (Local CLI)
- dbt Core Engine
- dbt-codegen Package
- Macros (macros/ and macros/helpers/)
- Vendored code (macros/vendored/dbt_core)
- Bash scripts (bash_scripts/base_model_creation.sh)
- Jinja Templating
- Database Adapters (Postgres, Snowflake, BigQuery, Redshift)
- File System (models/ directory, schema files)
- Integration Test Harness (integration_tests/)
- CI/CD Pipeline & Version Control
- GitHub Actions, CircleCI
- tox, Makefile
- Docker Compose

##### Relationships & Data Flow
- Developer runs `dbt run-operation` or invokes a macro in the IDE.
→ dbt Core loads project and dependencies (including dbt-codegen).
- dbt Core locates the appropriate macro in macros/
→ passes arguments into Jinja templates.
- Jinja engine compiles macro into SQL or YAML text.
→ For CLI output: returns generated code to stdout.
→ For file generation: the create_base_models macro calls the bash script
→ base_model_creation.sh writes files into the project’s models/ directory.
- Macros that introspect schema call dbt adapter methods
→ queries the target database metadata (via adapter layer).
- Integration tests spin up databases (via Docker Compose profiles), load seeds, run macros, and assert outputs.
- CI/CD (GitHub Actions / CircleCI) orchestrates linting (tox), integration tests, and release.

##### Architectural Patterns & Design Principles
- Plugin Architecture: dbt-codegen lives as a package that plugs into dbt’s macro registry.
- Separation of Concerns: SQL/Jinja macros (generate_*) are distinct from file-writing scripts.
- Adapter Pattern: Uses dbt’s adapter interface to support multiple warehouses.
- Test Isolation: integration_tests/ directory holds its own dbt_project.yml, seeds, models, and profiles for sandboxed validation.
- CI as Code: YAML workflows in .github/workflows and .circleci/config.yml automate validation across adapters.

- CLI → dbt Core
- dbt Core → Macros → Jinja → Output
- Macros → Adapter → Database
- create_base_models macro → Bash script → File System
- CI workflows → Integration Test Harness → Databases
- Core engine in another
- External systems (databases, CI) in a third

##### Detail Without Overthinking
- Major macro as its own component if it has distinct behavior (e.g., generate_source vs. generate_unit_test_template).
- Integration test suite and CI pipelines as first-class components.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Layer 1: Developer"
        Dev["Developer CLI / IDE"]:::user
        MF["Makefile"]:::tool
    end

    subgraph "Layer 2: dbt Core & Jinja"
        Core["dbt Core Engine"]:::core
        Jinja["Jinja Templating"]:::core
    end

    subgraph "Layer 3: dbt-codegen Plugin"
        subgraph "Macros"
            GSrc["generate_source.sql macro"]:::plugin
            GModelY["generate_model_yaml.sql macro"]:::plugin
            GUnit["generate_unit_test_template.sql macro"]:::plugin
            GBase["generate_base_model.sql macro"]:::plugin
            GImport["generate_model_import_ctes.sql macro"]:::plugin
            CreateBase["create_base_models.sql macro"]:::plugin
            Helpers["helpers.sql"]:::plugin
            Vendored["format_column.sql"]:::plugin
        end
        subgraph "Scripts"
            Bash["base_model_creation.sh script"]:::script
        end
    end

    subgraph "Layer 4: External Systems"
        subgraph Warehouses
            Postgres["Postgres Adapter"]:::external
            Snowflake["Snowflake Adapter"]:::external
            BigQuery["BigQuery Adapter"]:::external
            Redshift["Redshift Adapter"]:::external
        end
        DB["Database"]:::external
        FS["Project File System"]:::external
        subgraph "CI/CD"
            GH["GitHub Actions"]:::external
            Circle["CircleCI"]:::external
        end
        Env["supported_adapters.env"]:::external
    end

    subgraph "Layer 5: Tests"
        DC["docker-compose.yml"]:::tool
        IT["Integration Test Harness"]:::test
        ITMacros["Test Macros"]:::test
        ITOp["create_source_table.sql"]:::test
        ITModels["Test Models"]:::test
        ITSeeds["Test Seeds"]:::test
        ITTests["Test Cases"]:::test
        ITConfig["Test dbt Configs"]:::test
    end

    Dev -->|invokes via Makefile| MF
    MF -->|run-operation| Core
    Core -->|"loads macros"| GSrc
    Core -->|"loads macros"| GModelY
    Core -->|"loads macros"| GUnit
    Core -->|"loads macros"| GBase
    Core -->|"loads macros"| GImport
    Core -->|"loads macros"| CreateBase
    CreateBase -->|invokes| Bash
    Bash -->|writes files| FS
    GSrc -->|uses helpers| Helpers
    GBase -->|uses vendored| Vendored
    GSrc -->|jinja compile| Jinja
    GModelY -->|jinja compile| Jinja
    GUnit -->|jinja compile| Jinja
    GBase -->|jinja compile| Jinja
    GImport -->|jinja compile| Jinja
    Jinja -->|generated code| Dev
    Jinja -->|generated code| FS

    Core -->|adapter interface| Postgres
    Core -->|adapter interface| Snowflake
    Core -->|adapter interface| BigQuery
    Core -->|adapter interface| Redshift
    Postgres --> DB
    Snowflake --> DB
    BigQuery --> DB
    Redshift --> DB

    GH -->|orchestrates| MF
    Circle -->|orchestrates| MF

    GH -->|triggers| IT
    Circle -->|triggers| IT
    DC -->|profiles| IT
    IT -->|runs macros via Core| Core
    IT -->|validates data in| DB
    IT -->|checks outputs in| FS

    classDef user fill:#f9f,stroke:#333,stroke-width:1px
    classDef core fill:#bbf,stroke:#333,stroke-width:1px
    classDef plugin fill:#bfb,stroke:#333,stroke-width:1px
    classDef script fill:#fbf,stroke:#333,stroke-width:1px
    classDef external fill:#ffd,stroke:#333,stroke-width:1px
    classDef test fill:#fdd,stroke:#333,stroke-width:1px
    classDef tool fill:#ffb,stroke:#333,stroke-width:1px

    click Dev "https://github.com/dbt-labs/dbt-codegen/blob/main/run_test.sh"
    click MF "https://github.com/dbt-labs/dbt-codegen/tree/main/Makefile"
    click Core "https://github.com/dbt-labs/dbt-codegen/blob/main/dbt_project.yml"
    click GSrc "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/generate_source.sql"
    click GModelY "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/generate_model_yaml.sql"
    click GUnit "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/generate_unit_test_template.sql"
    click GBase "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/generate_base_model.sql"
    click GImport "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/generate_model_import_ctes.sql"
    click CreateBase "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/create_base_models.sql"
    click Helpers "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/helpers/helpers.sql"
    click Vendored "https://github.com/dbt-labs/dbt-codegen/blob/main/macros/vendored/dbt_core/format_column.sql"
    click Bash "https://github.com/dbt-labs/dbt-codegen/blob/main/bash_scripts/base_model_creation.sh"
    click Env "https://github.com/dbt-labs/dbt-codegen/blob/main/supported_adapters.env"
    click IT "https://github.com/dbt-labs/dbt-codegen/tree/main/integration_tests/"
    click ITMacros "https://github.com/dbt-labs/dbt-codegen/tree/main/integration_tests/macros/"
    click ITOp "https://github.com/dbt-labs/dbt-codegen/blob/main/integration_tests/macros/operations/create_source_table.sql"
    click ITModels "https://github.com/dbt-labs/dbt-codegen/tree/main/integration_tests/models/"
    click ITSeeds "https://github.com/dbt-labs/dbt-codegen/tree/main/integration_tests/seeds/"
    click ITTests "https://github.com/dbt-labs/dbt-codegen/tree/main/integration_tests/tests/"
    click ITConfig "https://github.com/dbt-labs/dbt-codegen/blob/main/integration_tests/dbt_project.yml"
    click GH "https://github.com/dbt-labs/dbt-codegen/blob/main/.github/workflows/ci.yml"
    click Circle "https://github.com/dbt-labs/dbt-codegen/blob/main/.circleci/config.yml"
    click DC "https://github.com/dbt-labs/dbt-codegen/blob/main/docker-compose.yml"
```
