# astronomer/astro-example-dags architecture

> Architecture diagram of the GitHub repository astronomer/astro-example-dags, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/astronomer/astro-example-dags
- Repository: https://github.com/astronomer/astro-example-dags

## Overview

##### Project Type and Purpose
- This is an Apache Airflow orchestration project scaffolded by Astronomer.
- Purpose: define, schedule, and run DAG-based ETL pipelines (e.g., the example “example_astronauts” DAG that fetches data from the Open Notify API).
- Key runtimes: multiple Docker containers running Airflow components plus a Postgres metadata database.

##### File Structure
- Top-level orchestration/config files:
- .astro/ (Astronomer project config and test scripts)
- Dockerfile / .dockerignore (build Astro Runtime image)
- .circleci/ and .github/workflows/ (CI/CD pipelines)
- packages.txt / requirements.txt (OS & Python deps)
- airflow_settings.yaml (local-only connections/variables/pools)
- Source of truth for pipelines:
- dags/ (Python DAG definitions)
- Tests:
- tests/dags/ (unit tests for DAG integrity)

##### Key Insights from the README
- Running locally with `astro dev start` spins up four Docker containers:
1. Postgres (metadata DB)
2. Airflow Webserver (UI)
3. Airflow Scheduler (task orchestration)
4. Airflow Triggerer (deferred task support)
- Exposes ports: 8080 (Webserver) and 5432 (Postgres).
- CI/CD: GitHub Actions / CircleCI deploy to Astronomer Cloud.

##### Main Components and Their Interactions
a. Developer / CI System
- Developer machine or CI pipeline pushes code & Dockerfile to GitHub.
- GitHub Actions / CircleCI build the Docker image, run tests, and deploy to Astronomer.
b. Astro Runtime / Docker
- Container image layers include the base Airflow image + your DAGs, plugins, OS/Python packages.
- Deployed to either local Docker or Astronomer Cloud.
c. Airflow Components (each in its own container)
1. Scheduler – reads DAG files, evaluates schedules & triggers tasks.
2. Triggerer – handles deferred tasks / sensors.
3. Webserver – serves the UI; shows DAG tree, logs, task status.
4. Worker (if using CeleryExecutor or KubernetesExecutor; in local dev default “SequentialExecutor” runs tasks in-process).
d. Postgres Metadata Database
- Stores DAG definitions, task states, variables, connections, XComs.
e. External Services
- Open Notify API (example DAG), any other sources/destinations your real DAGs will call.
f. Tests
- tests/dags/test_dag_integrity.py verifies DAG syntax and structure.

##### Architectural Patterns and Principles
- Containerized micro-services: each Airflow component isolated.
- Separation of concerns:
- DAG definitions separated from runtime engine & DB.
- Infrastructure (Dockerfiles, CI configs) separate from code.
- Declarative pipeline as code (DAG files + airflow_settings.yaml).
- CI/CD deployment pipeline for reproducible builds & automated tests.

##### Relationships & Data Flow
- CI/CD → (build & push) → Docker Registry / Astronomer
- Astro Runtime (Docker) → spawns containers
- Scheduler reads DAGs → writes to Postgres → triggers tasks
- Triggerer monitors deferred tasks → signals Scheduler
- Worker executes tasks → may call external API → logs results back to DB
- Webserver queries Postgres → serves UI to user

## Diagram (Mermaid)

```mermaid
graph TB
    %% CI/CD and Developer
    subgraph "Developer & CI/CD"
        direction TB
        Dev["Developer Machine"]:::external
        GitHub["GitHub Repo"]:::cd
        CircleCI["CircleCI Pipeline"]:::cd
        GHActions["GitHub Actions"]:::cd
        AstrCLI["Astronomer CLI"]:::cd
    end

    Dev -->|push code| GitHub
    GitHub -->|triggers| CircleCI
    GitHub -->|triggers| GHActions
    CircleCI -->|build/test/deploy| AstrCLI
    GHActions -->|deploy image| AstrCLI

    click GitHub "https://github.com/astronomer/astro-example-dags/blob/main/README.md"
    click CircleCI "https://github.com/astronomer/astro-example-dags/blob/main/.circleci/config.yml"
    click GHActions "https://github.com/astronomer/astro-example-dags/blob/main/.github/workflows/deploy-to-astro.yaml"
    click AstrCLI "https://github.com/astronomer/astro-example-dags/blob/main/.astro/config.yaml"
    click AstrCLI "https://github.com/astronomer/astro-example-dags/blob/main/.astro/test_dag_integrity_default.py"

    %% Docker Build Context
    subgraph "Docker Build Context"
        direction TB
        DockerfileNode["Dockerfile"]:::ci
        Dockerignore[".dockerignore"]:::ci
        Packages["packages.txt"]:::ci
        Requirements["requirements.txt"]:::ci
        AirflowSettings["airflow_settings.yaml"]:::config
        DAGSFolder["dags/"]:::code
        DAGSExample["example_astronauts.py"]:::code
        DAGIgnore[".airflowignore"]:::code
        TestsFolder["tests/dags/"]:::test
        TestFile["test_dag_integrity.py"]:::test
    end

    click DockerfileNode "https://github.com/astronomer/astro-example-dags/tree/main/Dockerfile"
    click Dockerignore "https://github.com/astronomer/astro-example-dags/blob/main/.dockerignore"
    click Packages "https://github.com/astronomer/astro-example-dags/blob/main/packages.txt"
    click Requirements "https://github.com/astronomer/astro-example-dags/blob/main/requirements.txt"
    click DAGSExample "https://github.com/astronomer/astro-example-dags/blob/main/dags/example_astronauts.py"
    click DAGIgnore "https://github.com/astronomer/astro-example-dags/blob/main/dags/.airflowignore"
    click TestFile "https://github.com/astronomer/astro-example-dags/blob/main/tests/dags/test_dag_integrity.py"
    click README "https://github.com/astronomer/astro-example-dags/blob/main/README.md"

    %% Docker Registry & Runtime
    AstrCLI -->|push image| Registry["Docker Registry / Astronomer Cloud"]:::external

    Registry -->|pull| Runtime["Astro Runtime (Docker)"]:::orchestration

    %% Airflow Containers
    subgraph "Airflow Containers" 
        direction TB
        Scheduler["Airflow Scheduler\n(port?)"]:::orchestration
        Triggerer["Airflow Triggerer"]:::orchestration
        Webserver["Airflow Webserver\n(port 8080)"]:::orchestration
        Worker["Airflow Worker"]:::orchestration
    end

    Runtime --> Scheduler
    Runtime --> Triggerer
    Runtime --> Webserver
    Runtime --> Worker

    %% Mounts & Config
    DAGSFolder -->|mount| Scheduler
    DAGSFolder -->|mount| Worker
    AirflowSettings -->|mount| Scheduler
    AirflowSettings -->|mount| Worker

    %% Postgres DB
    Postgres["Postgres Metadata DB\n(port 5432)"]:::db
    Scheduler ---|read/write| Postgres
    Triggerer ---|read/write| Postgres
    Webserver ---|read| Postgres
    Worker ---|read/write| Postgres

    %% External API
    API["Open Notify API"]:::external_green
    Worker -->|API call| API

    %% Styles
    classDef orchestration fill:#cce5ff,stroke:#00529b,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#666,stroke-width:1px
    classDef ci fill:#ffebcc,stroke:#b58900,stroke-width:1px
    classDef cd fill:#ffebcc,stroke:#b58900,stroke-width:1px
    classDef code fill:#e6ffe6,stroke:#2d662d,stroke-width:1px
    classDef test fill:#ffe6e6,stroke:#992d2d,stroke-width:1px
    classDef db fill:#eeeeee,stroke:#888888,stroke-width:1px
    classDef external_green fill:#e6ffee,stroke:#2d662d,stroke-width:1px
    classDef config fill:#f5f5f5,stroke:#999,stroke-width:1px
```
