# krishnaik06/etlweather architecture

> Architecture diagram of the GitHub repository krishnaik06/etlweather, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krishnaik06/etlweather
- Repository: https://github.com/krishnaik06/etlweather

## Overview

##### Project type + purpose (what you’re diagramming)
This is an **Astronomer/Astro Runtime Apache Airflow project**: a containerized Airflow deployment intended to run **DAG-defined data pipelines**. The “application” is not a traditional web app; the core deliverable is **workflow orchestration** (ETL/ELT) expressed as Python DAGs.

From the tree + README:
- Airflow runs locally via `astro dev start` / `docker-compose.yml`.
- Airflow is split into standard runtime components: **Webserver, Scheduler, Triggerer, Postgres metadata DB**.
- Business logic lives in `dags/` (e.g., `etlweather.py`, `exampledag.py`).
- Tests validate DAGs under `tests/dags/`.

##### CI/testing boundary (unit tests around DAG parsing/behavior)

Top-level artifacts and what each means architecturally:

- `docker-compose.yml`
- `Dockerfile`
- Defines the **Airflow runtime image** (Astro Runtime base + your additions). Show it as the build artifact feeding the containers.
- `airflow_settings.yaml`
- `requirements.txt` / `packages.txt`
- **Dependency inputs** into the image build; can affect operators/hooks/providers available to DAGs.
- `dags/etlweather.py`, `dags/exampledag.py`
- `tests/dags/test_dag_example.py`

##### What the README adds (runtime components + ports)
The README explicitly identifies the containers:
- **Postgres** = Airflow Metadata DB (port 5432)
- **Webserver** = UI/API (port 8080)
- **Scheduler** = schedules + triggers tasks
- **Triggerer** = deferred tasks (async sensors, etc.)

- Webserver ↔ Postgres (read/write metadata)
- Scheduler ↔ Postgres
- Triggerer ↔ Postgres

##### Developer/CI layer
- Developer machine / Git repo
- Test runner (pytest) executing `tests/dags/*`

##### Build layer
- Docker build using:
- `Dockerfile`
- `requirements.txt`
- `packages.txt`
- Output: “Custom Astro Runtime Image”

##### Runtime / Orchestration layer (Docker Compose)
- Airflow Webserver container
- Airflow Scheduler container
- Airflow Triggerer container
- Postgres (metadata DB) container

##### Pipeline definition layer
- `dags/etlweather.py`
- `dags/exampledag.py`
These should be shown as “loaded by Scheduler/Webserver from mounted DAGs folder”.

##### External systems layer (what DAGs talk to)
From README example: **Open Notify API** (for astronaut example)
From your repo hint: `etlweather.py` likely calls a weather API and may write somewhere.
- External API(s) (e.g., Weather API, Open Notify API)

**Control-plane (Airflow orchestration)**
- Webserver → Scheduler: “user triggers DAG / views status” (conceptual; not a direct call always, but via metadata)
- Webserver ↔ Postgres: “UI reads/writes DAG runs, task instances, connections”
- Scheduler ↔ Postgres: “scheduling state, task instance updates”
- Triggerer ↔ Postgres: “deferred task events/state”
- Scheduler → Task Execution: “runs tasks (executor)”

**Data-plane (ETL movement)**
- Task nodes → External APIs: “HTTP request”
- Task nodes → Data sink (if any): “write dataset”
- (Optional) Task nodes → XCom/metadata: “push/pull small control data via Airflow metadata DB” (only if DAG uses XComs; common in TaskFlow)

##### Architectural patterns/design principles present
- **Workflow orchestration pattern**: Airflow DAGs define dependencies; scheduler drives execution.
- **Containerized local dev**: Docker Compose encapsulates Airflow services.
- **Infrastructure-as-code-ish config**: `airflow_settings.yaml` for local bootstrapping of Connections/Variables.
- **Code-centric pipelines**: business logic is Python modules in `dags/`.
- **Testable DAGs**: `tests/dags/` suggests import/structure tests to prevent broken deployments.

##### Technologies
- “Astro Runtime (Apache Airflow)”
- “Docker / docker-compose”
- “Postgres (Airflow Metadata DB)”
- “Python DAGs (TaskFlow API likely used in example)”
- “External HTTP APIs”

##### Clear labels
- Name each container exactly: “airflow-webserver”, “airflow-scheduler”, “airflow-triggerer”, “postgres (metadata)”

- **Metadata/state traffic** (Airflow components ↔ Postgres)
- **Task execution/control** (Scheduler → Tasks)
- **Data movement** (Tasks → External systems)

##### Left: Source/config/build
- Repo → Docker build (Dockerfile + requirements/packages)
##### Center: Docker Compose boundary with 4 containers
- Airflow components to Postgres
- `dags/` volume mounted into webserver/scheduler
##### Right: External systems
- External API(s)
- External destinations (if present)

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% Legend
%% =========================
subgraph "Legend"
direction TB
L1["Airflow Runtime Service"]:::airflow
L2[("Metadata DB (Postgres)")]:::db
L3(["DAG / Pipeline Code"]):::dag
L4(["Task"]):::task
L5["Config / Spec File"]:::config
L6["External System (API / Sink)"]:::external
end

%% =========================
%% Source / Repo + Dev / CI
%% =========================
subgraph "Source Repo (Git Working Tree)"
direction TB
README["README (runtime components & ports)"]:::config

subgraph "Config & Dependency Inputs"
direction TB
DC["docker-compose.yml"]:::config
DF["Dockerfile"]:::config
AS["airflow_settings.yaml"]:::config
REQ["requirements.txt"]:::config
PKG["packages.txt"]:::config
end

subgraph "DAG Code"
direction TB
DAGS["dags/ (DAG definitions)"]:::dag
DAG_WEATHER["Weather ETL DAG"]:::dag
DAG_EXAMPLE["Example DAG"]:::dag
end

subgraph "Tests"
direction TB
TDIR["tests/dags/"]:::test
TEX["test_dag_example.py"]:::test
end
end

DEV["Developer / CI Runner (pytest)"]:::user
DEV -->|"runs_tests"| TDIR
TDIR -->|"imports_validates"| DAG_EXAMPLE
TDIR -->|"parses_loads"| DAGS

%% =========================
%% Build Layer
%% =========================
subgraph "Image Build (Astro Runtime + Customizations)"
direction TB
BUILD["Docker Build"]:::build
IMAGE["Custom Astro Runtime Image"]:::build
end

DF -->|"defines_image"| BUILD
REQ -->|"pip_deps"| BUILD
PKG -->|"apt_deps"| BUILD
BUILD -->|"produces"| IMAGE

%% =========================
%% Runtime / Deployment (Docker Compose)
%% =========================
subgraph "Local Runtime (Docker Compose)"
direction TB

subgraph "Airflow Platform (Astro Runtime)"
direction TB
WS["airflow-webserver (UI/API :8080)"]:::airflow
SCH["airflow-scheduler"]:::airflow
TRG["airflow-triggerer"]:::airflow
EXEC["Task Execution (within Airflow runtime)"]:::airflow
end

META[("postgres (Airflow metadata) :5432")]:::db

end

DC -->|"defines_topology"| WS
DC -->|"defines_topology"| SCH
DC -->|"defines_topology"| TRG
DC -->|"defines_topology"| META

IMAGE -->|"container_image"| WS
IMAGE -->|"container_image"| SCH
IMAGE -->|"container_image"| TRG

AS -->|"bootstraps_connections_variables_pools"| META

%% Control-plane metadata/state
WS <-->|"metadata_rw"| META
SCH <-->|"metadata_rw"| META
TRG <-->|"metadata_rw"| META

%% Orchestration flow
WS -->|"trigger_view_runs_via_metadata"| META
SCH -->|"schedules_and_queues_tasks"| EXEC
TRG -->|"resumes_deferred_tasks"| EXEC

%% DAG loading (mounted code)
DAGS -->|"mounted_loaded"| WS
DAGS -->|"mounted_loaded"| SCH
DAGS -->|"mounted_loaded"| TRG
DAG_WEATHER -->|"dag_definition"| SCH
DAG_EXAMPLE -->|"dag_definition"| SCH

%% =========================
%% DAG-level Workflow View
%% =========================
subgraph "Pipelines (DAG Execution Dataflow)"
direction TB

subgraph "etlweather.py (Weather ETL)"
direction TB
W_T1("Extract Weather Data (HTTP)"):::task
W_T2("Transform / Validate"):::task
W_T3("Load / Persist (TBD)"):::task
W_T1 -->|"dataflow"| W_T2 -->|"dataflow"| W_T3
end

subgraph "exampledag.py (Astronaut Example)"
direction TB
E_T1("Fetch Astronauts (Open Notify API)"):::task
E_T2("Process / Publish Result"):::task
E_T1 -->|"dataflow"| E_T2
end

end

%% Scheduler -> DAG tasks (control)
SCH -->|"executes_tasks"| W_T1
SCH -->|"executes_tasks"| E_T1

%% Optional metadata / XCom boundary
W_T2 -->|"small_state_xcom_metadata"| META
E_T2 -->|"small_state_xcom_metadata"| META

%% =========================
%% External Systems
%% =========================
subgraph "External Systems"
direction TB
WEATHER_API["Weather API (external HTTP)"]:::external
OPEN_API["Open Notify API (external HTTP)"]:::external
SINK["External Destination (TBD by DAG code)"]:::externalTBD
end

W_T1 -->|"http_request"| WEATHER_API
E_T1 -->|"http_request"| OPEN_API
W_T3 -->|"write_dataset"| SINK

%% =========================
%% Click Events (component_mapping paths)
%% =========================
click DC "https://github.com/krishnaik06/etlweather/blob/main/docker-compose.yml"
click DF "https://github.com/krishnaik06/etlweather/tree/main/Dockerfile"
click AS "https://github.com/krishnaik06/etlweather/blob/main/airflow_settings.yaml"
click REQ "https://github.com/krishnaik06/etlweather/blob/main/requirements.txt"
click PKG "https://github.com/krishnaik06/etlweather/blob/main/packages.txt"
click DAGS "https://github.com/krishnaik06/etlweather/tree/main/dags/"
click DAG_WEATHER "https://github.com/krishnaik06/etlweather/blob/main/dags/etlweather.py"
click DAG_EXAMPLE "https://github.com/krishnaik06/etlweather/blob/main/dags/exampledag.py"
click TDIR "https://github.com/krishnaik06/etlweather/tree/main/tests/dags/"
click TEX "https://github.com/krishnaik06/etlweather/blob/main/tests/dags/test_dag_example.py"
click README "https://github.com/krishnaik06/etlweather/blob/main/README.md"

%% =========================
%% Styles
%% =========================
classDef airflow fill:#2B6CB0,stroke:#1A365D,color:#FFFFFF,stroke-width:1px
classDef db fill:#ECC94B,stroke:#975A16,color:#1A202C,stroke-width:1px
classDef dag fill:#2F855A,stroke:#22543D,color:#FFFFFF,stroke-width:1px
classDef task fill:#C6F6D5,stroke:#2F855A,color:#1A202C,stroke-width:1px
classDef config fill:#EDF2F7,stroke:#4A5568,color:#1A202C,stroke-width:1px
classDef build fill:#9F7AEA,stroke:#553C9A,color:#FFFFFF,stroke-width:1px
classDef external fill:#E53E3E,stroke:#742A2A,color:#FFFFFF,stroke-width:1px
classDef externalTBD fill:#FED7D7,stroke:#E53E3E,color:#1A202C,stroke-width:1px,stroke-dasharray: 5 3
classDef test fill:#BEE3F8,stroke:#2C5282,color:#1A202C,stroke-width:1px
classDef user fill:#718096,stroke:#2D3748,color:#FFFFFF,stroke-width:1px
```
