# elliotgunn/datafold-dbt-ci architecture

> Architecture diagram of the GitHub repository elliotgunn/datafold-dbt-ci, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elliotgunn/datafold-dbt-ci
- Repository: https://github.com/elliotgunn/datafold-dbt-ci

## Overview

##### Project Type and Purpose
- This is a dbt (data build tool) project designed to illustrate and automate a CI pipeline. Its purpose is to automatically compile dbt models, run tests against them, and lint SQL code changes.
- The repository includes standard dbt components such as models, seeds, snapshots, and macros, along with GitHub Actions configuration for running the CI jobs (compilation, tests, linting).

##### Main Components
a. GitHub Actions CI Pipeline:
- .github/workflows configuration (e.g., dbt-pr-ci-job.yml) as the CI orchestrator.
- That the pipeline is triggered by Pull Request events and automatically executes three key steps: model compilation, testing, and SQL linting.
b. dbt Components:
- Models: Separate the models directory into components. For instance, show the "staging" area (which includes sub-folders like jaffle_shop and stripe) and the "marts" area (with core models such as dim_customers.sql and fct_orders.sql).
- Seeds: Display the seed data source containing CSV files and related YAML files (customers, orders, payments) used to simulate raw input data.
- Macros: Include the macros directory that houses Jinja templates or utility macros used in model transformations.
- Snapshots & Tests: Illustrate the snapshots directory and test files (like tests/assert_stg_payments_amount_is_positive.sql) that verify data consistency and quality.
c. External and Supporting Components:
- External Data Sources: Although the project uses seed files (using fake data from a fictional app and jaffle_shop), a component representing the external data source/data ingestion point can be depicted.
- Tools & Dependencies: Include items such as SQLFluff (for linting), dbt framework, and Python requirements (requirements.txt) that support the project execution.
- Repository Trigger: Indicate the external trigger (i.e., a new PR event) initiating the CI workflow.

##### Relationships and Interactions
- A PR is opened → triggers the GitHub Actions workflow.
- The workflow then:
1. Runs dbt to compile and build models.
2. Executes tests to validate the data models.
3. Executes a SQL linter (SQLFluff) against the changes.

##### Architectural Pattern and Design Principles
- The design follows a modular structure, where different folders (models, seeds, macros, tests) encapsulate distinct responsibilities.
- The CI-driven workflow embodies an automated testing culture — changes are verified for compilation, correctness, and conformance to SQL standards.
- It is built as a fully integrated pipeline where code changes pass through multiple stages before being merged, ensuring data quality and compliance.

## Diagram (Mermaid)

```mermaid
flowchart TD
    PR["Pull Request Event"]:::external
    CI["GitHub Actions CI Pipeline"]:::ci

    PR -->|"triggers"| CI

    subgraph "CI Pipeline Steps"
        COMP["dbt Compile/Build Models"]:::ci
        TEST["Run Tests"]:::test
        LINT["SQL Linting (SQLFluff)"]:::ci
        COMP --> TEST
        TEST --> LINT
    end

    CI -->|"executes"| COMP
    LINT -->|"reportsStatus"| STATUS["Status Report to PR"]:::external

    subgraph "dbt Components"
        SEEDS["Seeds"]:::seeds
        STAGING["Models (Staging)"]:::dtt
        MARTS["Models (Marts)"]:::dtt
        MACROS["Macros"]:::macro
        SNAPSHOTS["Snapshots"]:::external
        TESTS["Tests"]:::test
    end

    COMP -->|"compilesUsing"| STAGING
    STAGING -->|"transformsTo"| MARTS
    SEEDS -->|"feedsDataTo"| STAGING
    STAGING -->|"validatedBy"| TESTS

    subgraph "Supporting Tools"
        TOOLS["Tools & Dependencies"]:::tool
    end

    TOOLS -->|"configures"| STAGING
    TOOLS --> MARTS
    TOOLS --> MACROS

    %% Click Events
    click CI "https://github.com/elliotgunn/datafold-dbt-ci/blob/main/.github/workflows/dbt-pr-ci-job.yml"
    click STAGING "https://github.com/elliotgunn/datafold-dbt-ci/tree/main/models/staging"
    click MARTS "https://github.com/elliotgunn/datafold-dbt-ci/tree/main/models/marts"
    click SEEDS "https://github.com/elliotgunn/datafold-dbt-ci/tree/main/seeds"
    click MACROS "https://github.com/elliotgunn/datafold-dbt-ci/tree/main/macros"
    click SNAPSHOTS "https://github.com/elliotgunn/datafold-dbt-ci/tree/main/snapshots"
    click TESTS "https://github.com/elliotgunn/datafold-dbt-ci/blob/main/tests/assert_stg_payments_amount_is_positive.sql"
    click LINT "https://github.com/elliotgunn/datafold-dbt-ci/blob/main/.sqlfluff"
    click TOOLS "https://github.com/elliotgunn/datafold-dbt-ci/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef ci fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef dtt fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef seeds fill:#ffeeba,stroke:#856404,stroke-width:2px;
    classDef macro fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef test fill:#f5c6cb,stroke:#721c24,stroke-width:2px;
    classDef tool fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
