# ogsiji/airflow-tutorial architecture

> Architecture diagram of the GitHub repository ogsiji/airflow-tutorial, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ogsiji/airflow-tutorial
- Repository: https://github.com/ogsiji/airflow-tutorial

## Overview

Based on our analysis of both the file tree and the README, this project is clearly a full-fledged data pipeline system tailored for retail analytics. It makes heavy use of Apache Airflow (using Astronomer’s CLI and runtime) to orchestrate workflows (via DAGs), integrates dbt for transforming and modeling data, leverages Soda for data quality checks, and interacts with cloud services such as Google Cloud Storage and BigQuery.

##### Main Components
- Airflow Environment:
- DAGs Folder (“dags”): Contains the primary workflows (e.g., example_dag_basic, example_dag_advanced, retail.py).
- Airflow Core Containers: Scheduler, Webserver, Postgres (metadata database), and Triggerer (as spun up by Astro CLI).
- Configuration Files: Dockerfile, docker-compose.override.yml, airflow_settings.yaml that show containerization and environment setup.
- Data Transformation and Modeling:
- dbt Files: Located under the “include/dbt” directory with its project configuration (dbt_project.yml) and models (report, sources, transform).
- Tests and Logs: dbt target folders (compiled models, run logs) that provide insight into model builds.
- Data Quality Checks:
- Soda Folder: Under “include/soda” containing checks configuration and Python scripts to perform quality checks.
- Data Sources/External Integrations:
- Cloud Integrations: “gcp” folder for Google Cloud-specific configurations.
- Data Files: “include/dataset” (like online_retail.csv) for sample or raw data.
- Metabase Dashboard: “include/metabase-data” indicates visualization or BI tools integration.
- Supporting Files:
- Packages and Requirements: packages.txt and requirements.txt indicate OS-level and Python dependencies.
- Tests: “tests” directory (e.g., tests/dags) for testing Airflow DAG integrity.
- Docker and Astro-related Files: Dockerfile and docker-compose files support containerized and local development.

##### Representing Relationships and Data Flow
- For the orchestration (Airflow), show its interaction with:
- The Metadata Database (Postgres),
- The Webserver UI (user interface for monitoring the DAGs), and
- The Scheduler/Triggerer which launch tasks.
- How individual DAG tasks interface with other components:
- Tasks that trigger dbt jobs (data transformation/modeling),
- Tasks running Soda scripts (data quality checks),
- Tasks orchestrating data ingestion (upload with Astro SDK to BigQuery via Google Cloud Storage).
- External connections such as:
- Cloud services (GCS and BigQuery),
- Metabase for BI/dashboards,
- Possibly integrations with other services via APIs.

##### Architectural Patterns and Design Principles
- ETL Pipeline and Modular DAG Design: Each DAG represents a standalone workflow that taps on different functionalities (extraction, transformation with dbt, quality assurance with Soda, and data load).
- Separation of Concerns: Distinguish between orchestration (handled by Airflow and its supporting files), data transformation (managed via dbt), quality management (using Soda), and external integrations (GCP and Metabase).

##### Relevant Technologies, Frameworks, and Libraries
- Apache Airflow (with its components like Scheduler, Webserver, Triggerer).
- Astro CLI and Docker (indicating containerized environments).
- dbt (for transformation and modeling; also highlighting its own configuration and compiled models).
- Soda (for data quality checks).
- Google Cloud Platform (GCS and BigQuery integration).
- Metabase (for dashboarding and visual analytics).
- Additional libraries as noted in requirements.txt or packages.txt.

- Clear labels for each component (for example, label “Airflow Scheduler”, “Airflow Webserver”, “dbt Transformation”, “Soda Data Quality”, “Postgres Metadata DB”, “Google Cloud Storage”, “BigQuery”, “Metabase Dashboard”).
- Annotations: Briefly annotate interactions or dependencies (e.g., “Triggers dbt transformation”, “Validates data quality”, “Uploads to GCS”, “Ingests into BigQuery”).

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Airflow Orchestration"
        A1["Airflow Scheduler"]:::airflow
        A2["Airflow Webserver"]:::airflow
        A3["Airflow Triggerer"]:::airflow
        A4["Metadata DB (Postgres)"]:::database
        DAGS["DAGs Folder"]:::folder
        CON["Container & Env Config"]:::configuration
    end

    subgraph "Data Transformation"
        DBT_Task["dbt Transformation Task"]:::dbt
        DBT_Project["dbt Project"]:::dbt
        DBT_Models["dbt Models"]:::dbt
    end

    subgraph "Data Quality"
        SODA["Soda Data Quality"]:::soda
    end

    subgraph "External Integrations"
        GCP["Google Cloud Platform"]:::cloud
        GCS["Google Cloud Storage"]:::cloud
        BQ["BigQuery"]:::database
        METABASE["Metabase Dashboard"]:::metabase
        RAWDATA["Raw Data (CSV)"]:::folder
    end

    subgraph "Supporting Files"
        PACKAGES["packages.txt"]:::support
        REQUIREMENTS["requirements.txt"]:::support
        TESTS["Airflow DAG Tests"]:::support
    end

    %% Orchestration Layer Connections
    CON --> A1
    CON --> A2
    CON --> A3
    CON --> A4
    A1 --> DAGS
    TESTS --> DAGS

    %% Airflow triggers tasks
    A1 -->|"triggers_dbt"| DBT_Task
    A1 -->|"triggers_soda"| SODA

    %% Data Pipeline Flow
    RAWDATA -->|"feeds_into"| DBT_Task
    DBT_Task -->|"runs_project"| DBT_Project
    DBT_Project -->|"executes_models"| DBT_Models
    DBT_Task -->|"uploads_to"| GCS
    GCP -->|"configures"| GCS
    GCS -->|"loads_into"| BQ
    BQ -->|"serves_for"| METABASE

    %% Click Events for Component Mapping
    click DAGS "https://github.com/ogsiji/airflow-tutorial/tree/main//dags"
    click CON "https://github.com/ogsiji/airflow-tutorial/tree/main//Dockerfile"
    click CON "https://github.com/ogsiji/airflow-tutorial/blob/main//docker-compose.override.yml"
    click CON "https://github.com/ogsiji/airflow-tutorial/blob/main//.astro/config.yaml"
    click DBT_Project "https://github.com/ogsiji/airflow-tutorial/tree/main//include/dbt"
    click DBT_Models "https://github.com/ogsiji/airflow-tutorial/tree/main//include/dbt/models"
    click SODA "https://github.com/ogsiji/airflow-tutorial/tree/main//include/soda"
    click GCP "https://github.com/ogsiji/airflow-tutorial/tree/main//include/gcp"
    click RAWDATA "https://github.com/ogsiji/airflow-tutorial/tree/main//include/dataset"
    click METABASE "https://github.com/ogsiji/airflow-tutorial/tree/main//include/metabase-data"
    click PACKAGES "https://github.com/ogsiji/airflow-tutorial/blob/main//packages.txt"
    click REQUIREMENTS "https://github.com/ogsiji/airflow-tutorial/blob/main//requirements.txt"
    click TESTS "https://github.com/ogsiji/airflow-tutorial/tree/main//tests/dags"

    %% Styles
    classDef airflow fill:#bfd8d2,stroke:#333,stroke-width:2px;
    classDef database fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef dbt fill:#d4e6f1,stroke:#333,stroke-width:2px;
    classDef soda fill:#fad7a0,stroke:#333,stroke-width:2px;
    classDef cloud fill:#d5f5e3,stroke:#333,stroke-width:2px;
    classDef metabase fill:#f5b7b1,stroke:#333,stroke-width:2px;
    classDef folder fill:#e8daef,stroke:#333,stroke-width:2px;
    classDef support fill:#f5cba7,stroke:#333,stroke-width:2px;
    classDef configuration fill:#ffe6e6,stroke:#333,stroke-width:2px;
```
