How astronomer/astro-example-dags works
Project Type and Purpose
• This is an Apache Airflow orchestration project scaffolded by Astronomer.
• Purpose: define, schedule, and run DAG-based ETL pipelines (e.g., the example “example_astronauts” DAG that fetches data from the Open Notify API).
• Key runtimes: multiple Docker containers running Airflow components plus a Postgres metadata database.
File Structure
• Top-level orchestration/config files:
• .astro/ (Astronomer project config and test scripts)
• Dockerfile / .dockerignore (build Astro Runtime image)
• .circleci/ and .github/workflows/ (CI/CD pipelines)
• packages.txt / requirements.txt (OS & Python deps)
• airflow_settings.yaml (local-only connections/variables/pools)
• Source of truth for pipelines:
• dags/ (Python DAG definitions)
• Tests:
• tests/dags/ (unit tests for DAG integrity)
Key Insights from the README
• Running locally with astro dev start spins up four Docker containers:
1. Postgres (metadata DB)
2. Airflow Webserver (UI)
3. Airflow Scheduler (task orchestration)
4. Airflow Triggerer (deferred task support)
• Exposes ports: 8080 (Webserver) and 5432 (Postgres).
• CI/CD: GitHub Actions / CircleCI deploy to Astronomer Cloud.
Main Components and Their Interactions
a. Developer / CI System
• Developer machine or CI pipeline pushes code & Dockerfile to GitHub.
• GitHub Actions / CircleCI build the Docker image, run tests, and deploy to Astronomer.
b. Astro Runtime / Docker
• Container image layers include the base Airflow image + your DAGs, plugins, OS/Python packages.
• Deployed to either local Docker or Astronomer Cloud.
c. Airflow Components (each in its own container)
1. Scheduler – reads DAG files, evaluates schedules & triggers tasks.
2. Triggerer – handles deferred tasks / sensors.
3. Webserver – serves the UI; shows DAG tree, logs, task status.
4. Worker (if using CeleryExecutor or KubernetesExecutor; in local dev default “SequentialExecutor” runs tasks in-process).
d. Postgres Metadata Database
• Stores DAG definitions, task states, variables, connections, XComs.
e. External Services
• Open Notify API (example DAG), any other sources/destinations your real DAGs will call.
f. Tests
• tests/dags/test_dag_integrity.py verifies DAG syntax and structure.
Architectural Patterns and Principles
• Containerized micro-services: each Airflow component isolated.
• Separation of concerns:
• DAG definitions separated from runtime engine & DB.
• Infrastructure (Dockerfiles, CI configs) separate from code.
• Declarative pipeline as code (DAG files + airflow_settings.yaml).
• CI/CD deployment pipeline for reproducible builds & automated tests.
Relationships & Data Flow
• CI/CD → (build & push) → Docker Registry / Astronomer
• Astro Runtime (Docker) → spawns containers
• Scheduler reads DAGs → writes to Postgres → triggers tasks
• Triggerer monitors deferred tasks → signals Scheduler
• Worker executes tasks → may call external API → logs results back to DB
• Webserver queries Postgres → serves UI to user