# aliyun/dbt-maxcompute architecture

> Architecture diagram of the GitHub repository aliyun/dbt-maxcompute, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aliyun/dbt-maxcompute
- Repository: https://github.com/aliyun/dbt-maxcompute

## Overview

##### Project type and purpose
- This is a Python-based adapter plugin for dbt Core, named **dbt-maxcompute**. Its purpose is to extend dbt’s ELT framework so that users can run dbt models on Alibaba MaxCompute.
- It implements the dbt “adapter” interface (connection management, compilation, materializations) and provides MaxCompute-specific macros for SQL materialization.

##### File structure
- Top-level:
- `dbt/adapters/maxcompute/` — Python adapter implementation (connections.py, impl.py, relation.py, etc.)
- `dbt/include/maxcompute/` — Jinja SQL macros and materializations for MaxCompute
- `tests/` — Unit and functional tests covering adapter and macro behavior
- `docs/` — User and developer documentation (Tutorial, authentication)
- CI config under `.github/workflows`
- `setup.py`, `dev-requirements.txt` — package and dev setup

##### README for technical insights
- Confirms this is an alpha release of a dbt adapter for MaxCompute v1.8.0.
- Describes authentication options, adapter-specific config parameters, and required profile settings.
- Mentions reliance on MaxCompute REST/tunnel endpoints (via PyODPS or the adapter) to submit and fetch queries.

##### Final recommendations
- Be very detailed: break out sub-modules (ConnectionManager vs. AdapterImpl vs. Macros).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    LocalConfig["Local Config & Profiles"]:::config
    click LocalConfig "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/include/maxcompute/profile_template.yml"

    %% Core Component
    dbtCore["dbt Core"]:::core

    %% Plugin Layer
    subgraph "dbt-maxcompute Plugin" 
        direction TB
        CM["Connection Manager"]:::plugin
        click CM "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/connections.py"

        subgraph "Adapter Implementation"
            direction TB
            impl["impl.py"]:::plugin
            click impl "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/impl.py"
            relation["relation.py"]:::plugin
            click relation "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/relation.py"
            column["column.py"]:::plugin
            click column "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/column.py"
            context["context.py"]:::plugin
            click context "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/context.py"
        end

        subgraph "Credential & Config Parsers"
            direction TB
            cred["credentials.py"]:::plugin
            click cred "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/credentials.py"
            setting["setting_parser.py"]:::plugin
            click setting "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/setting_parser.py"
        end

        subgraph "Utility Layer & Wrapper"
            direction TB
            utils["utils.py"]:::plugin
            click utils "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/utils.py"
            wrapper["wrapper.py"]:::plugin
            click wrapper "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/wrapper.py"
        end

        subgraph "Relation Configs"
            direction TB
            rc_base["_base.py"]:::plugin
            click rc_base "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/relation_configs/_base.py"
            rc_mv["_materialized_view.py"]:::plugin
            click rc_mv "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/relation_configs/_materialized_view.py"
            rc_part["_partition.py"]:::plugin
            click rc_part "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/relation_configs/_partition.py"
            rc_pol["_policies.py"]:::plugin
            click rc_pol "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/adapters/maxcompute/relation_configs/_policies.py"
        end
    end

    %% Jinja Macros
    JinjaMacros["Jinja Macros & SQL Templates"]:::macro
    click JinjaMacros "https://github.com/aliyun/dbt-maxcompute/blob/master/dbt/include/maxcompute/macros/adapters.sql"

    %% PyODPS SDK & Tests
    PyODPS["PyODPS SDK"]:::external

    subgraph "PyODPS Tests"
        direction TB
        t1["get_table_test.py"]:::external
        click t1 "https://github.com/aliyun/dbt-maxcompute/blob/master/tests/pyodps/get_table_test.py"
        t2["timestamp_test.py"]:::external
        click t2 "https://github.com/aliyun/dbt-maxcompute/blob/master/tests/pyodps/timestamp_test.py"
    end

    %% External Service
    MaxCompute["Alibaba MaxCompute Service"]:::service

    %% CI and Package Setup
    CI["CI Config (GitHub Actions)"]:::config
    click CI "https://github.com/aliyun/dbt-maxcompute/blob/master/.github/workflows/main.yml"
    Setup["Package & Dev Setup"]:::config
    click Setup "https://github.com/aliyun/dbt-maxcompute/blob/master/setup.py"

    %% Interactions
    LocalConfig -->|"loads profiles"| dbtCore
    dbtCore -->|"register adapter"| CM
    dbtCore -->|"compile models"| JinjaMacros
    CM -->|"uses credentials"| cred
    CM -->|"manages connection"| PyODPS
    impl -->|"implements adapter API"| dbtCore
    JinjaMacros -->|"render SQL"| impl
    impl -->|"call utils"| utils
    impl -->|"use relation configs"| rc_base
    CM -->|"submit SQL"| PyODPS
    PyODPS -->|"HTTP/Tunnel"| MaxCompute
    MaxCompute -->|"results"| PyODPS
    PyODPS -->|"return results"| CM
    CM -->|"report status"| dbtCore

    %% Testing
    PyODPS -->|"tested by"| t1
    PyODPS -->|"tested by"| t2

    %% CI and Setup flows
    Setup --> CI
    CI --> dbtCore

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef plugin fill:#E6FFEA,stroke:#1B7F3A,color:#1B7F3A;
    classDef macro fill:#FFF5CC,stroke:#D9A300,color:#D9A300;
    classDef external fill:#F0F0F0,stroke:#999999,color:#333333;
    classDef service fill:#FFE5CC,stroke:#D97706,color:#B45309;
    classDef config fill:#E8F0FE,stroke:#4285F4,color:#1967D2;
```
