# rockingspq/dbt-snowflake-airflow architecture

> Architecture diagram of the GitHub repository rockingspq/dbt-snowflake-airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rockingspq/dbt-snowflake-airflow
- Repository: https://github.com/rockingspq/dbt-snowflake-airflow

## Overview

Based on a careful review of the project’s file tree and README, this project is an Apache Airflow-based workflow orchestrated using Astronomer. It not only contains standard Airflow DAG files (shown in the top-level “dags” folder with examples such as “exampledag.py”) but also integrates a DBT project (located under “dags/data_pipeline”) for managing data transformation models, tests, and even SQL macros. Essentially, the project is a data pipeline combining orchestration (via Airflow) with analytics/data modeling (via DBT), deployed as an Astronomer project using Docker containers.

- The Scheduler, Webserver, and Triggerer, highlighting the orchestration flow within Airflow.
- The Airflow Metadata Database (Postgres), indicating that all components interact with persistent metadata storage.
- The Executor/DAGs area that reads DAG scripts from the “dags” folder.
- For the DBT side, indicate that a specific Airflow DAG (likely the “dbt_dag.py”) triggers the DBT transformations. Show how task execution in Airflow invokes DBT commands which then use models, seeds, macros, and tests.
- How Docker acts as the containerized environment that assembles these components together for local (or Astronomer cloud) deployment.

##### Architectural Patterns / Design Principles
- Modular Separation: Separate the orchestration layer (Airflow) from the transformation layer (DBT) while showing explicit interactions between them.
- Containerization: Emphasize the use of Docker to encapsulate Airflow components along with dependency management via packages.txt and requirements.txt.
- Data Pipeline Layering: Within the DBT project, identify “staging” (raw data ingestion and cleansing) versus “marts” (aggregated analytics), as well as the use of macros to centralize business logic.
- Extensibility through Plugins: Although the “plugins” directory is empty, mention that it’s the designated extension point for custom Airflow functionality.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Environment"
        A_Docker["Docker Container (Astro Runtime)"]:::docker

        %% Airflow Control Plane
        subgraph "Control Plane"
            A_Webserver["Airflow Webserver (port 8080)"]:::airflow
            A_Scheduler["Airflow Scheduler"]:::airflow
            A_Trigger["Airflow Triggerer"]:::airflow
            A_Postgres["Postgres Metadata DB (port 5432)"]:::database
        end

        %% Airflow Data Pipeline
        subgraph "Data Plane"
            A_DAGs["Airflow DAGs Folder"]:::dag
            %% DBT Data Transformation
            subgraph "DBT Data Transformation"
                DBT_Project["DBT Project"]:::dbt
                subgraph "DBT Layers"
                    DBT_Staging["DBT - Staging Models"]:::dbt
                    DBT_Marts["DBT - Marts Models"]:::dbt
                    DBT_Macros["DBT - Macros"]:::dbt
                    DBT_Tests["DBT - Tests/Seeds/Snapshots"]:::dbt
                end
            end
        end

        %% Deployment Configurations
        subgraph "Deployment Config"
            Dockerfile["Dockerfile"]:::dockerconf
            Packages["packages.txt"]:::dockerconf
            Requirements["requirements.txt"]:::dockerconf
        end
    end

    %% Airflow Interactions
    A_Scheduler -->|"reads_DAGs"| A_DAGs
    A_Trigger -->|"communicates_with"| A_Webserver
    A_Scheduler -->|"communicates_with"| A_Webserver
    A_Webserver -->|"accesses"| A_Postgres
    A_Scheduler -->|"accesses"| A_Postgres
    A_Trigger -->|"accesses"| A_Postgres

    %% Orchestration to DBT trigger
    A_DAGs -->|"triggers_DBT"| DBT_Project

    %% DBT internal data flow
    DBT_Project -->|"utilizes"| DBT_Staging
    DBT_Project -->|"utilizes"| DBT_Marts
    DBT_Project -->|"utilizes"| DBT_Macros
    DBT_Project -->|"utilizes"| DBT_Tests

    %% Deployment config associations
    Dockerfile -->|"configures"| A_Docker
    Packages -->|"installs_on"| A_Docker
    Requirements -->|"installs_on"| A_Docker

    %% Click Events
    click A_DAGs "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/"
    click DBT_Project "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/data_pipeline/"
    click DBT_Staging "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/data_pipeline/models/staging/"
    click DBT_Marts "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/data_pipeline/models/marts/"
    click DBT_Macros "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/data_pipeline/macros/"
    click DBT_Tests "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/dags/data_pipeline/tests/"
    click Dockerfile "https://github.com/rockingspq/dbt-snowflake-airflow/tree/main/Dockerfile"
    click Packages "https://github.com/rockingspq/dbt-snowflake-airflow/blob/main/packages.txt"
    click Requirements "https://github.com/rockingspq/dbt-snowflake-airflow/blob/main/requirements.txt"

    %% Styles
    classDef airflow fill:#ffebcd,stroke:#b22222,stroke-width:2px;
    classDef dbt fill:#e0ffff,stroke:#008b8b,stroke-width:2px;
    classDef docker fill:#e6ffe6,stroke:#006400,stroke-width:2px;
    classDef database fill:#fffacd,stroke:#daa520,stroke-width:2px;
    classDef dockerconf fill:#e0e0ff,stroke:#00008b,stroke-width:2px;
    classDef dag fill:#fff0f5,stroke:#c71585,stroke-width:2px;
```
