How starburstdata/dbt-trino-utils works
1. Repository purpose
- trino_utils is a dbt package that provides Trino/Starburst-compatible macro implementations.
- Its main role is to act as a dispatch layer: when another dbt package calls a macro, dbt can route to trino_utils first for Trino-specific SQL, then fall back to the upstream package.
2. Core architectural boundary
- The package is not a standalone application or service.
- The main boundary is between:
- consumer dbt projects
- this package’s macros
- upstream packages (dbt_utils, dbt_date, and dbt_metrics)
- the target warehouse: Trino or Starburst
3. Primary code surface: macros
- The macros/ directory contains the actual compatibility implementations.
- It is organized by upstream package namespace:
- macros/dbt_utils/
- macros/dbt_date/
- macros/dbt_metrics/
- These macros define Trino/Starburst-specific SQL behavior for functions like date spine generation, deduplication, generic tests, timezone conversion, and metric SQL generation.
4. Compatibility model
- This repo is a set of “shims” rather than a full fork.
- The important architectural idea is that a consuming dbt project enables dispatch in dbt_project.yml, so macro resolution searches trino_utils before the upstream namespace.
- That means this repo only overrides the parts of upstream packages that need warehouse-specific SQL.
5. Upstream package linkage
- The repository includes git submodules for:
- dbt-utils
- dbt-date
- dbt_metrics
- These submodules are useful as a reference/compatibility source and help keep the shim implementations aligned with upstream package behavior.
6. Macro categories
- The macros fall into three functional groups:
- SQL generation helpers for Trino-compatible query patterns
- generic tests adapted for Trino semantics
- schema cleanup / maintenance operations for dev database hygiene
- Cleanup macros are operational utilities rather than model logic, used via dbt run-operation.
7. Integration test architecture
- The integration_tests/ directory contains separate dbt projects for validating each supported upstream package behavior:
- integration_tests/dbt_utils
- integration_tests/dbt_date
- integration_tests/dbt_metrics
- Each test project packages trino_utils and exercises the dispatched macros in realistic dbt flows using models, seeds, and tests.
8. Data flow in tests
- Test data typically flows as:
- CSV seed files or test models in the integration project
- dbt runs invoking macros from trino_utils
- generated SQL executed against a Trino/Starburst instance
- results compared to expected output or schema assertions
- This validates that the shimmed macros behave correctly in warehouse-specific execution.
9. Warehouse/runtime environment
- The repo includes Docker assets for local and CI execution against both:
- Trino
- Starburst
- docker-compose-trino.yml and docker-compose-starburst.yml define those environments.
- The docker/ directory contains warehouse config, init/removal scripts, and a dbt container image used to run integration tests.
10. CI and automation
- GitHub Actions (.github/workflows/ci.yml) runs the repository’s validation pipeline.
- The CI setup appears centered on spinning up the containerized warehouse, running dbt integration tests, and verifying macro compatibility across supported packages.
11. dbt project structure
- The root dbt_project.yml makes this repository behave as a dbt package.
- The package layout is intentionally macro-heavy: there are no primary analytical models at the root, because the product is the macro library itself.
12. Operational utilities
- The cleanup macros in macros/dbt_utils/schema_cleanup/ are notable because they support developer workflow around Trino schemas:
- drop schemas by prefix
- drop a schema by name
- drop relations no longer present in the project
- These are invoked as dbt operations, not as part of model builds.
13. How to diagram it
- A useful architecture diagram would show:
- consumer dbt project on the left
- trino_utils in the middle as the dispatch/override layer
- upstream packages as fallback references
- Trino/Starburst as the execution target on the right
- integration test projects and Dockerized warehouse setup as a validation path beneath the main flow
14. Overall design takeaway
- The repo’s architecture is intentionally small and focused: it does not own data pipelines itself.
- Its job is to make existing dbt package macros portable to Trino/Starburst by providing a thin, warehouse-aware SQL compatibility layer and testing that layer in isolated dbt projects.
Main entry points: dbt_project.yml trino_utils · dbt_utils/dbt_project.yml dbt_utils tests · trino Trino · ci.yml CI workflow
Components 18
Core package
Dispatch layer and compatibility macros
- Consumer dbt · A downstream project enabling dispatch to resolve macros through trino_utils first
- trino_utils · Root dbt package entrypoint that defines this repository as a macro library
dbt_project.yml - dbt_utils SQL · Trino-compatible implementations for upstream dbt_utils SQL helpers
macros/dbt_utils/sql - dbt_utils tests · Generic tests adapted for Trino semantics
macros/dbt_utils/generic_tests - Schema cleanup · dbt run-operation utilities for dropping schemas and stale relations
macros/dbt_utils/schema_cleanup - dbt_date · Timezone conversion compatibility for dbt_date
macros/dbt_date/convert_timezone.sql - dbt_metrics SQL · Metric SQL generation adapted for Trino and Starburst
macros/dbt_metrics/sql_gen
Upstream refs
Referenced packages mirrored by the shims
- dbt-utils · Referenced upstream package used as compatibility target and fallback
dbt-utils - dbt-date · Referenced upstream date package used as compatibility target and fallback
dbt-date - dbt_metrics · Referenced upstream metrics package used as compatibility target and fallback
dbt_metrics
Validation
Integration test projects and CI
- dbt_utils tests · dbt project that exercises dbt_utils dispatch behavior against Trino
integration_tests/dbt_utils/dbt_project.yml - dbt_date tests · dbt project that validates dbt_date compatibility macros
integration_tests/dbt_date/dbt_project.yml - dbt_metrics tests · dbt project that validates metric SQL generation and calendar joins
integration_tests/dbt_metrics/dbt_project.yml - CI workflow · GitHub Actions pipeline that runs containerized integration validation
.github/workflows/ci.yml
Runtime
Containerized Trino and Starburst environments
- Docker compose · Compose entrypoints for local and CI execution on Trino and Starburst
docker-compose-trino.yml - Docker assets · Warehouse configs and dbt image used to initialize and run tests
docker - Trino · Execution target for generated SQL and dispatched macros
docker/trino - Starburst · Alternative execution target for generated SQL and dispatched macros
docker/starburst
Connections 24
- Consumer dbt → trino_utils: dispatches to · Downstream dbt projects route macro resolution into this package first
- trino_utils → dbt_utils SQL: exposes · Package root publishes the Trino-compatible dbt_utils SQL overrides
- trino_utils → dbt_utils tests: exposes · Package root publishes generic test overrides for dbt_utils
- trino_utils → Schema cleanup: exposes · Package root includes operational cleanup macros for dbt run-operation
- trino_utils → dbt_date: exposes · Package root includes dbt_date compatibility macros
- trino_utils → dbt_metrics SQL: exposes · Package root includes dbt_metrics SQL generation macros
- dbt_utils SQL → dbt-utils: matches · Implementations track upstream dbt-utils behavior while changing warehouse-specific SQL
- dbt_utils tests → dbt-utils: matches · Test overrides preserve upstream semantics under Trino
- dbt_date → dbt-date: matches · Timezone conversion shim aligns with the upstream package contract
- dbt_metrics SQL → dbt_metrics: matches · Metric SQL generation follows the upstream dbt_metrics interface
- dbt_utils tests → dbt_utils SQL: validates · Integration project executes deduplication and related dbt_utils macros
- dbt_utils tests → dbt_utils tests: validates · Integration project runs generic tests against Trino semantics
- dbt_date tests → dbt_date: validates · Integration project checks date and timezone compatibility behavior
- dbt_metrics tests → dbt_metrics SQL: validates · Integration project exercises metric SQL generation and calendar joins
- CI workflow → dbt_utils tests: runs · CI executes the dbt_utils integration project
- CI workflow → dbt_date tests: runs · CI executes the dbt_date integration project
- CI workflow → dbt_metrics tests: runs · CI executes the dbt_metrics integration project
- Docker compose → Trino: starts · Compose file provisions the Trino warehouse for tests
- Docker compose → Starburst: starts · Compose file provisions the Starburst warehouse for tests
- Docker assets → Trino: configures · Warehouse config and init scripts prepare the Trino runtime
- Docker assets → Starburst: configures · Warehouse config and init scripts prepare the Starburst runtime
- dbt_utils tests → Trino: executes on · Integration dbt runs generate SQL against Trino
- dbt_date tests → Trino: executes on · Integration dbt runs generate SQL against Trino
- dbt_metrics tests → Starburst: executes on · Integration dbt runs generate SQL against Starburst or Trino-compatible warehouse targets