# androchentw/airflow-playground architecture

> Architecture diagram of the GitHub repository androchentw/airflow-playground, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/androchentw/airflow-playground
- Repository: https://github.com/androchentw/airflow-playground

## Overview

- Running a standalone Airflow hello-world,
- Setting up a Great Expectations hello-world,
- Combining Great Expectations with Airflow via DAGs, and
- Integrating Great Expectations with GCP BigQuery.

##### Main Components
- Airflow Environment: Represent the Apache Airflow orchestration system with key subcomponents like the scheduler, webserver, worker(s), and the Docker container(s) these run in (given the docker-compose.yaml).
- DAGs: Show a specific component for the DAGs (located in the “dags” directory). Mark the “great_expectations_airflow.py” DAG that integrates Great Expectations validations within Airflow.
- Great Expectations: Illustrate the role of Great Expectations, as demonstrated by the examples in the “src” directory (both the notebook and Python script). This component handles data quality validations.
- GCP BigQuery (optional): Since one of the topics is Great Expectations + GCP BigQuery, include an external component for the BigQuery data warehouse that may serve as a target or source of data validations.
- Docker Environment: Since docker-compose.yaml is used, include a Docker container layer to represent how the system environment is spun up.
- Repository/Source Files: Mention sample notebooks and scripts from “src” and configuration files that support the orchestration and validations.

##### Relationships and Interactions
- Data flow paths: From ingestion (if applicable) into Airflow, moving through DAG tasks to data validations, and then possibly outputting to BigQuery or storing results.
- Docker as a foundational layer that encapsulates the Airflow and Great Expectations components.

##### Architectural Patterns and Design Principles
- A modular separation between orchestration (Airflow) and data validation (Great Expectations), each with its own clearly defined responsibilities.
- The system is containerized (using docker-compose), so represent this abstraction to underline how environment consistency is maintained.

##### Relevant Technologies, Frameworks, and Libraries
- Apache Airflow – for orchestration (scheduler, webserver, workers)
- Great Expectations – for data validation
- Docker & Docker Compose – for containerization and environment setup
- (Optional) GCP BigQuery – for the external data warehouse
- Python scripts, Jupyter notebooks, and possibly the airflow-provider-great-expectations package, as mentioned in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Compose Environment"
        DC["Docker Compose File"]:::docker
        subgraph "Airflow Environment"
            A1["Airflow Scheduler"]:::airflow
            A2["Airflow Webserver"]:::airflow
            A3["Airflow Worker"]:::airflow
            A4["Airflow DAG (great_expectations_airflow.py)"]:::airflow
        end
        subgraph "Great Expectations"
            GE1["GE Script (great_expectations_hello_world.py)"]:::ge
            GE2["GE Notebook (great_expectations_airflow.py)"]:::ge
            %% Note: Reusing the GE Notebook component to show alternative example.
            %% Adjust GE2 label if needed to reflect the actual file name.
            GE2["GE Notebook (great_expectations_hello_world.ipynb)"]:::ge
        end
    end

    BQ["BigQuery Data Warehouse"]:::bigquery
    Docs1["Project README (README.md)"]:::docs
    Docs2["GE Docs (src/README.md)"]:::docs

    %% Connections between Airflow components
    A1 --> A4
    A2 --> A4
    A3 --> A4

    %% Airflow triggers Great Expectations validations
    A4 -->|"triggers"| GE1
    A4 -->|"triggers"| GE2

    %% Great Expectations validates data with BigQuery
    GE1 -->|"validates"| BQ
    GE2 -->|"validates"| BQ

    %% Documentation supports the components
    Docs1 -->|"supports"| A4
    Docs2 -->|"supports"| GE1

    %% Docker Compose file drives the Airflow Environment
    DC --> A1

    %% Class definitions for styling
    classDef airflow fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef ge fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef docker fill:#577590,stroke:#fff,stroke-width:2px,color:#fff;
    classDef bigquery fill:#f94144,stroke:#333,stroke-width:2px;
    classDef docs fill:#f3722c,stroke:#333,stroke-width:2px;

    %% Click Events
    click DC "https://github.com/androchentw/airflow-playground/blob/main/docker-compose.yaml"
    click A4 "https://github.com/androchentw/airflow-playground/blob/main/dags/great_expectations_airflow.py"
    click GE1 "https://github.com/androchentw/airflow-playground/blob/main/src/great_expectations_hello_world.py"
    click GE2 "https://github.com/androchentw/airflow-playground/blob/main/src/great_expectations_hello_world.ipynb"
    click Docs1 "https://github.com/androchentw/airflow-playground/blob/main/README.md"
    click Docs2 "https://github.com/androchentw/airflow-playground/blob/main/src/README.md"
```
