How insight-services-apac/apac-capability-dai-dbtfabricsparknb works
Project type and purpose
• This is a Python-based dbt adapter plugin (“dbt-fabricsparknb”) plus a thin “dbt_wrapper” library that lets dbt run against Microsoft Fabric Spark notebooks without a Livy or SQL endpoint.
• It integrates with dbt-core as an “adapter” package, and wraps calls into Fabric’s Spark-notebook REST APIs and session management.
File structure
• Top-level Python package modules:
• dbt/adapters/fabricsparknb – the adapter implementation, connections, catalog, manifest, notebook orchestration, credentials, utilities
• dbt_wrapper – helper library that handles API calls, SQL construction, file generation, pipeline configuration, stage execution
• Configuration & packaging:
• pyproject.toml, Makefile, scripts/ for build, CI/CD, devcontainer/ for VS Code remote containers
• tests/unit and tests/functional exercise adapter behaviors under various scenarios
• mkdocs.yml drives site generation
README
• Provides project overview: “dbt-fabricsparknb” (Insight version) built on community “dbt-fabricspark”
• Points to hosted GH Pages for complete docs, but confirms core responsibility: let dbt work with Fabric Spark without external endpoint
How to model this
a. Main components
• Developer workstation (CLI environment / IDE)
• dbt CLI + plugins (dbt-core + dbt-fabricsparknb adapter)
• dbt_wrapper module
• Microsoft Fabric REST API (Spark Notebook service)
• Spark execution engine (Fabric Spark cluster)
• Lakehouse / underlying storage (Delta tables, metadata)
• Local metadata store (DuckDB catalog, dbt manifest/cache)
• CI/CD pipeline (GitHub Actions workflows, devcontainer)
b. Relationships & interactions
1. Developer issues dbt commands (e.g., dbt run).
2. dbt-core loads the “fabricsparknb” adapter plugin.
3. Adapter calls into dbt_wrapper to prepare SQL, notebook cells, and pipeline config.
4. dbt_wrapper uses Fabric REST API endpoints (auth via fabric_spark_credentials) to spin up Livy-style Spark sessions or Notebook sessions.
5. The Spark engine executes model SQL against the Lakehouse storage.
6. Results (manifests, catalogs, run results) flow back to local dbt metadata store (DuckDB + JSON files).
7. CI/CD workflows test the adapter and deploy new versions to PyPI or Fabric Workspace.
c. Architectural patterns & design principles
• Plugin architecture (dbt adapter hook points)
• Layered design: adapter layer vs. wrapper HTTP/API layer vs. core dbt
• Separation of concerns: SQL generation, session management, result parsing, metadata handling
• Config-driven: profiles.yml, pipeline_config.py, Makefile targets, mkdocs.yml
d. Key technologies & libraries
• Python 3.x, dbt-core (adapter interface)
• Requests or Fabric SDK for REST API integration
• DuckDB (local catalog)
• CI/CD: GitHub Actions, pre-commit, pytest, flake8
• Devcontainer (Docker + VS Code Remote)
Diagramting conventions
• CLI → adapter → wrapper → Fabric API → Spark
• Spark → Lakehouse (read/write)
• Fabric API → adapter (job status, catalog)
• Adapter → Local Metadata
• GitHub Actions → Codebase → PyPI/Workspace
• Developer tools (blue), adapter code (green), external services (orange), storage (brown), CI/CD (purple)