How krishnaik06/etlweather works
Project type + purpose (what you’re diagramming)
This is an Astronomer/Astro Runtime Apache Airflow project: a containerized Airflow deployment intended to run DAG-defined data pipelines. The “application” is not a traditional web app; the core deliverable is workflow orchestration (ETL/ELT) expressed as Python DAGs.
From the tree + README:
• Airflow runs locally via astro dev start / docker-compose.yml.
• Airflow is split into standard runtime components: Webserver, Scheduler, Triggerer, Postgres metadata DB.
• Business logic lives in dags/ (e.g., etlweather.py, exampledag.py).
• Tests validate DAGs under tests/dags/.
CI/testing boundary (unit tests around DAG parsing/behavior)
Top-level artifacts and what each means architecturally:
• docker-compose.yml
• Dockerfile
• Defines the Airflow runtime image (Astro Runtime base + your additions). Show it as the build artifact feeding the containers.
• airflow_settings.yaml
• requirements.txt / packages.txt
• Dependency inputs into the image build; can affect operators/hooks/providers available to DAGs.
• dags/etlweather.py, dags/exampledag.py
• tests/dags/test_dag_example.py
What the README adds (runtime components + ports)
The README explicitly identifies the containers:
• Postgres = Airflow Metadata DB (port 5432)
• Webserver = UI/API (port 8080)
• Scheduler = schedules + triggers tasks
• Triggerer = deferred tasks (async sensors, etc.)
• Webserver ↔ Postgres (read/write metadata)
• Scheduler ↔ Postgres
• Triggerer ↔ Postgres
Developer/CI layer
• Developer machine / Git repo
• Test runner (pytest) executing tests/dags/*
Build layer
• Docker build using:
• Dockerfile
• requirements.txt
• packages.txt
• Output: “Custom Astro Runtime Image”
Runtime / Orchestration layer (Docker Compose)
• Airflow Webserver container
• Airflow Scheduler container
• Airflow Triggerer container
• Postgres (metadata DB) container
Pipeline definition layer
• dags/etlweather.py
• dags/exampledag.py
These should be shown as “loaded by Scheduler/Webserver from mounted DAGs folder”.
External systems layer (what DAGs talk to)
From README example: Open Notify API (for astronaut example)
From your repo hint: etlweather.py likely calls a weather API and may write somewhere.
• External API(s) (e.g., Weather API, Open Notify API)
Control-plane (Airflow orchestration)
• Webserver → Scheduler: “user triggers DAG / views status” (conceptual; not a direct call always, but via metadata)
• Webserver ↔ Postgres: “UI reads/writes DAG runs, task instances, connections”
• Scheduler ↔ Postgres: “scheduling state, task instance updates”
• Triggerer ↔ Postgres: “deferred task events/state”
• Scheduler → Task Execution: “runs tasks (executor)”
Data-plane (ETL movement)
• Task nodes → External APIs: “HTTP request”
• Task nodes → Data sink (if any): “write dataset”
• (Optional) Task nodes → XCom/metadata: “push/pull small control data via Airflow metadata DB” (only if DAG uses XComs; common in TaskFlow)
Architectural patterns/design principles present
• Workflow orchestration pattern: Airflow DAGs define dependencies; scheduler drives execution.
• Containerized local dev: Docker Compose encapsulates Airflow services.
• Infrastructure-as-code-ish config: airflow_settings.yaml for local bootstrapping of Connections/Variables.
• Code-centric pipelines: business logic is Python modules in dags/.
• Testable DAGs: tests/dags/ suggests import/structure tests to prevent broken deployments.
Technologies
• “Astro Runtime (Apache Airflow)”
• “Docker / docker-compose”
• “Postgres (Airflow Metadata DB)”
• “Python DAGs (TaskFlow API likely used in example)”
• “External HTTP APIs”
Clear labels
• Name each container exactly: “airflow-webserver”, “airflow-scheduler”, “airflow-triggerer”, “postgres (metadata)”
• Metadata/state traffic (Airflow components ↔ Postgres)
• Task execution/control (Scheduler → Tasks)
• Data movement (Tasks → External systems)
Left: Source/config/build
• Repo → Docker build (Dockerfile + requirements/packages)
Center: Docker Compose boundary with 4 containers
• Airflow components to Postgres
• dags/ volume mounted into webserver/scheduler
Right: External systems
• External API(s)
• External destinations (if present)