pavan19047/dbt-core

1. This is a Rust monorepo for dbt Core v2 / Fusion. The architecture is centered around shared libraries plus a small set of binaries: dbt-main is the main CLI/runtime entrypoint, with specialized tools like dbt-docs-server, dbt-metricflow, dbt-sa-cli, dbt-login, dbt-ci, and dbt-adbc.

2. The first major subsystem is project loading and validation. dbt-loader, dbt-parser, dbt-profile, dbt-cloud-config, dbt-schemas, and dbt-selector-parser read dbt projects from the filesystem (dbt_project.yml, profiles.yml, SQL/YAML resources, packages, selectors) and resolve them into typed internal structures. This is the boundary where user-authored project files become normalized dbt metadata.

3. Jinja rendering is a distinct layer, not embedded in the parser itself. The dbt-jinja-* crates provide context building, variables, filters, template evaluation, and phase-specific environments for parse/compile/run. They also implement the language surface that dbt exposes to model SQL and macros.

4. Compilation and planning are handled by dbt-compilation, dbt-dag, dbt-scheduler, dbt-defer, dbt-metadata, dbt-metadata-parquet, dbt-schema-store, and dbt-index-core. Together these crates build the DAG, hydrate schemas, determine dependencies, compute scheduling, and produce the artifact graph used downstream by execution, docs, and lineage features.

5. The execution/runtime layer is split from compilation. dbt-tasks-sa and dbt-tasks-core model runnable work units such as models, seeds, snapshots, tests, and operations. They convert compiled nodes into task execution, manage task context, and bridge into warehouse execution and run-cache behavior.

6. Database connectivity is abstracted through the adapter/engine stack in dbt-adapter and dbt-adbc. This is where dbt crosses the boundary to external warehouses and engines such as Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB, Spark, ClickHouse, Fabric, Salesforce, Athena, and Exasol. The adapter layer also handles SQL generation, relation naming, metadata queries, retries, query comments, sidecar communication, and ADBC driver management.

7. Authentication and connection setup are separate concerns. dbt-auth and dbt-platform-auth provide credential resolution for warehouse/cloud integrations, including env-based config, cloud YAML, browser/OAuth flows, session caching, and service-specific auth helpers. dbt-cloud-api is the typed HTTP client for dbt Cloud.

8. Artifact and state management is a major architectural boundary. dbt-state, dbt-common, dbt-error, dbt-base, and dbt-metadata-parquet handle cancellation, state comparison, run caches, metadata persistence, error modeling, and serialization. The repo leans heavily on Parquet for scalable artifacts while still maintaining JSON compatibility for older consumers.

9. Local documentation is served by dbt-docs-server. It is not just a static site server; it exposes handlers for lineage, models, seeds, snapshots, tests, metrics, searches, exposures, macros, files, identities, and project data, all backed by the parsed/compiled artifacts and state stores.

10. Observability is another cross-cutting subsystem. dbt-tracing, dbt-telemetry, dbt-tui-progress, and tracing support inside dbt-common emit structured logs, spans, metrics, JSONL/Parquet/OTLP telemetry, and terminal progress. This instrumentation spans parsing, compilation, execution, and cloud interactions.

11. The repo also contains support libraries for domain-specific concerns: dbt-agate and dbt-csv for table/CSV handling, dbt-classification-types for taxonomy and tagging, dbt-features and dbt-clap-core for feature gating and CLI plumbing, dbt-env and dbt-ident for environment and identifier handling, and dbt-frontend-common for shared SQL/expression/type utilities.

12. Testing and reproducibility are built into the architecture. dbt-test-utils, dbt-test-containers, and many crate-local snapshot/integration tests provide deterministic harnesses, including containerized Postgres and warehouse-adjacent fixtures. Record/replay support appears in adbc-record-replay and the time-machine/trace subsystems to make warehouse interactions testable without always hitting live services.

13. The overall flow is: project files and auth config are loaded from disk and env, Jinja and parsing resolve them into typed resources, compilation turns resources into a DAG and artifacts, tasks execute against warehouse adapters, and the resulting run state/metadata/telemetry feeds docs, lineage, caches, and analytics surfaces.