# insight-services-apac/apac-capability-dai-dbtfabricsparknb architecture

> Architecture diagram of the GitHub repository insight-services-apac/apac-capability-dai-dbtfabricsparknb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb
- Repository: https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb

## Overview

##### Project type and purpose
- This is a Python-based dbt adapter plugin (“dbt-fabricsparknb”) plus a thin “dbt_wrapper” library that lets dbt run against Microsoft Fabric Spark notebooks without a Livy or SQL endpoint.
- It integrates with dbt-core as an “adapter” package, and wraps calls into Fabric’s Spark-notebook REST APIs and session management.

##### File structure
- Top-level Python package modules:
- dbt/adapters/fabricsparknb – the adapter implementation, connections, catalog, manifest, notebook orchestration, credentials, utilities
- dbt_wrapper – helper library that handles API calls, SQL construction, file generation, pipeline configuration, stage execution
- Configuration & packaging:
- pyproject.toml, Makefile, scripts/ for build, CI/CD, devcontainer/ for VS Code remote containers
- tests/unit and tests/functional exercise adapter behaviors under various scenarios
- mkdocs.yml drives site generation

##### README
- Provides project overview: “dbt-fabricsparknb” (Insight version) built on community “dbt-fabricspark”
- Points to hosted GH Pages for complete docs, but confirms core responsibility: let dbt work with Fabric Spark without external endpoint

##### How to model this
a. Main components
- Developer workstation (CLI environment / IDE)
- dbt CLI + plugins (dbt-core + dbt-fabricsparknb adapter)
- dbt_wrapper module
- Microsoft Fabric REST API (Spark Notebook service)
- Spark execution engine (Fabric Spark cluster)
- Lakehouse / underlying storage (Delta tables, metadata)
- Local metadata store (DuckDB catalog, dbt manifest/cache)
- CI/CD pipeline (GitHub Actions workflows, devcontainer)
b. Relationships & interactions
1. Developer issues dbt commands (e.g., dbt run).
2. dbt-core loads the “fabricsparknb” adapter plugin.
3. Adapter calls into dbt_wrapper to prepare SQL, notebook cells, and pipeline config.
4. dbt_wrapper uses Fabric REST API endpoints (auth via fabric_spark_credentials) to spin up Livy-style Spark sessions or Notebook sessions.
5. The Spark engine executes model SQL against the Lakehouse storage.
6. Results (manifests, catalogs, run results) flow back to local dbt metadata store (DuckDB + JSON files).
7. CI/CD workflows test the adapter and deploy new versions to PyPI or Fabric Workspace.
c. Architectural patterns & design principles
- Plugin architecture (dbt adapter hook points)
- Layered design: adapter layer vs. wrapper HTTP/API layer vs. core dbt
- Separation of concerns: SQL generation, session management, result parsing, metadata handling
- Config-driven: profiles.yml, pipeline_config.py, Makefile targets, mkdocs.yml
d. Key technologies & libraries
- Python 3.x, dbt-core (adapter interface)
- Requests or Fabric SDK for REST API integration
- DuckDB (local catalog)
- CI/CD: GitHub Actions, pre-commit, pytest, flake8
- Devcontainer (Docker + VS Code Remote)

##### Diagramting conventions
- CLI → adapter → wrapper → Fabric API → Spark
- Spark → Lakehouse (read/write)
- Fabric API → adapter (job status, catalog)
- Adapter → Local Metadata
- GitHub Actions → Codebase → PyPI/Workspace
- Developer tools (blue), adapter code (green), external services (orange), storage (brown), CI/CD (purple)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Environment
    subgraph "Developer Environment"
        Dev["Developer Workstation\n(CLI/IDE)"]:::devtools
        CLI["dbt CLI + plugins"]:::devtools
    end
    Dev --> CLI

    %% Adapter Layer
    subgraph "Adapter Layer"
        A_init["__init__.py"]:::adapter
        A_version["__version__.py"]:::adapter
        A_conn["connections.py"]:::adapter
        A_cred["fabric_spark_credentials.py"]:::adapter
        A_cat["catalog.py"]:::adapter
        A_manifest["manifest.py"]:::adapter
        A_notebook["notebook.py"]:::adapter
        A_livy["livysession.py"]:::adapter
        A_impl["impl.py"]:::adapter
        A_mock["mock.py"]:::adapter
        A_utilsA["utils.py"]:::adapter
    end
    CLI --> A_init
    A_init --> A_impl

    %% Wrapper Layer
    subgraph "Wrapper Layer"
        W_main["main.py"]:::wrapper
        W_workflows["workflows.py"]:::wrapper
        W_api["fabric_api.py"]:::wrapper
        W_sql["fabric_sql.py"]:::wrapper
        W_gen["generate_files.py"]:::wrapper
        W_pconfig["pipeline_config.py"]:::wrapper
        W_stage["stage_executor.py"]:::wrapper
        W_wrapper["wrapper.py"]:::wrapper
        W_log["log_levels.py"]:::wrapper
        W_hash["hashcheck_levels.py"]:::wrapper
        W_catW["catalog.py"]:::wrapper
        W_utilsW["utils.py"]:::wrapper
    end
    A_impl --> W_main
    W_main --> W_api
    W_main --> W_sql
    W_main --> W_gen
    W_main --> W_pconfig
    W_main --> W_stage

    %% External Services
    subgraph "External Services"
        API["Fabric Spark Notebook API"]:::external
        Spark["Spark Execution Engine"]:::external
    end
    W_api --> API
    API --> Spark

    %% Storage
    subgraph "Storage"
        Lakehouse["Lakehouse Storage\n(Delta Tables)"]:::storage
        LocalMeta["Local Metadata Store\n(DuckDB & JSON)"]:::storage
    end
    Spark --> Lakehouse
    API --> LocalMeta
    A_impl --> LocalMeta

    %% CI/CD & Development Container
    subgraph "CI/CD & DevContainer"
        CI["GitHub Actions Workflows"]:::cicd
        DevCont["Devcontainer Config"]:::cicd
    end
    CI --> A_init
    CI --> W_main
    DevCont --> Dev

    %% Click Events
    click A_init "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/__init__.py"
    click A_version "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/__version__.py"
    click A_conn "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/connections.py"
    click A_cred "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/fabric_spark_credentials.py"
    click A_cat "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/catalog.py"
    click A_manifest "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/manifest.py"
    click A_notebook "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/notebook.py"
    click A_livy "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/livysession.py"
    click A_impl "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/impl.py"
    click A_mock "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/mock.py"
    click A_utilsA "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt/adapters/fabricsparknb/utils.py"

    click W_main "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/main.py"
    click W_workflows "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/workflows.py"
    click W_api "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/fabric_api.py"
    click W_sql "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/fabric_sql.py"
    click W_gen "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/generate_files.py"
    click W_pconfig "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/pipeline_config.py"
    click W_stage "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/stage_executor.py"
    click W_wrapper "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/wrapper.py"
    click W_log "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/log_levels.py"
    click W_hash "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/hashcheck_levels.py"
    click W_catW "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/catalog.py"
    click W_utilsW "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/dbt_wrapper/utils.py"

    click CI "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/tree/main/.github/workflows/"
    click DevCont "https://github.com/insight-services-apac/apac-capability-dai-dbtfabricsparknb/blob/main/.devcontainer/devcontainer.json"

    %% Styles
    classDef devtools fill:#D0E8FF,stroke:#0284C7,color:#000
    classDef adapter fill:#DFFFE0,stroke:#16A34A,color:#000
    classDef wrapper fill:#E0F2FE,stroke:#3B82F6,color:#000
    classDef external fill:#FFE4B5,stroke:#C2410C,color:#000
    classDef storage fill:#F7DF9B,stroke:#92400E,color:#000
    classDef cicd fill:#F5E1FF,stroke:#9333EA,color:#000
```
