# fmiles-2006/ingdatos-clima-sidney-grupo06 architecture

> Architecture diagram of the GitHub repository fmiles-2006/ingdatos-clima-sidney-grupo06, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fmiles-2006/ingdatos-clima-sidney-grupo06
- Repository: https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06

## Overview

This is a daily Sydney-weather data pipeline: Open-Meteo data is incrementally ingested into Delta bronze, transformed and quality-checked by dbt, published to PostgreSQL, and served to Metabase. Airflow orchestrates the runtime workflow. The graph also retains the documented Grafana observability capability. dbt model-level connections follow the README’s documented lineage; unsampled model internals are not inferred beyond that.

Main entry points: [`dags/pipeline_grupo06.py`](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dags/pipeline_grupo06.py) (Pipeline DAG), [`dbt/models/staging/stg_clima_diario.sql`](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_clima_diario.sql) (Daily climate staging), [`scripts/publicar_gold.py`](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/publicar_gold.py) (Gold publisher), [`scripts/ingesta.py`](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/ingesta.py) (Weather ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_ingesta["Weather ingestion<br/>[ingesta.py]"]
  node_bronze[("Delta bronze")]
end

subgraph group_transform["Transformation"]
  node_stg_clima["Daily climate staging"]
  node_stg_codigos["Weather code staging"]
  node_stg_enso["ENSO staging<br/>[stg_fases_enso.sql]"]
  node_monthly["Monthly climate facts"]
  node_daily_features["Daily climate features"]
  node_dbt_quality["dbt quality checks"]
  node_dbt_runner["dbt project"]
end

subgraph group_serving["Serving"]
  node_publisher["Gold publisher<br/>[publicar_gold.py]"]
  node_duckdb[("DuckDB warehouse")]
  node_postgres[("PostgreSQL gold")]
  node_metabase(("Metabase dashboard"))
  node_grafana(("Grafana monitoring"))
end

subgraph group_orchestration["Orchestration"]
  node_airflow_dag["Pipeline DAG"]
end

node_open_meteo(("Open-Meteo"))

node_airflow_dag -->|"runs"| node_ingesta
node_open_meteo -->|"provides weather"| node_ingesta
node_ingesta -->|"writes Delta"| node_bronze
node_airflow_dag -->|"runs dbt"| node_dbt_runner
node_bronze -->|"feeds staging"| node_stg_clima
node_stg_clima -->|"builds marts"| node_monthly
node_stg_clima -->|"builds marts"| node_daily_features
node_airflow_dag -->|"runs tests"| node_dbt_quality
node_dbt_runner -->|"builds models"| node_duckdb
node_airflow_dag -->|"runs publication"| node_publisher
node_publisher -->|"reads gold"| node_duckdb
node_publisher -->|"publishes tables"| node_postgres
node_postgres -->|"serves dashboard"| node_metabase
node_grafana -.->|"monitors runs"| node_airflow_dag

click node_ingesta "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/ingesta.py"
click node_stg_clima "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_clima_diario.sql"
click node_stg_codigos "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_codigos_clima.sql"
click node_stg_enso "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_fases_enso.sql"
click node_monthly "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/marts/fct_clima_mensual.sql"
click node_daily_features "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/marts/feat_clima_diario.sql"
click node_dbt_quality "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/tree/main/dbt/tests"
click node_dbt_runner "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/tree/main/dbt"
click node_airflow_dag "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dags/pipeline_grupo06.py"
click node_publisher "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/publicar_gold.py"
click node_grafana "https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/docs/grafana/observabilidad_grupo06.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingesta,node_bronze toneBlue
class node_stg_clima,node_stg_codigos,node_stg_enso,node_monthly,node_daily_features,node_dbt_quality,node_dbt_runner toneAmber
class node_publisher,node_duckdb,node_postgres,node_metabase,node_grafana toneMint
class node_airflow_dag toneRose
class node_open_meteo toneIndigo
```

## Components

### Ingestion

- [Weather ingestion](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/ingesta.py): `scripts/ingesta.py`
- **Delta bronze**

### Transformation

- [Daily climate staging](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_clima_diario.sql): `dbt/models/staging/stg_clima_diario.sql`
- [Weather code staging](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_codigos_clima.sql): `dbt/models/staging/stg_codigos_clima.sql`
- [ENSO staging](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/staging/stg_fases_enso.sql): `dbt/models/staging/stg_fases_enso.sql`
- [Monthly climate facts](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/marts/fct_clima_mensual.sql): `dbt/models/marts/fct_clima_mensual.sql`
- [Daily climate features](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dbt/models/marts/feat_clima_diario.sql): `dbt/models/marts/feat_clima_diario.sql`
- [dbt quality checks](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/tree/main/dbt/tests): `dbt/tests`
- [dbt project](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/tree/main/dbt): `dbt`

### Serving

- [Gold publisher](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/scripts/publicar_gold.py): `scripts/publicar_gold.py`
- **DuckDB warehouse**
- **PostgreSQL gold**
- **Metabase dashboard**
- [Grafana monitoring](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/docs/grafana/observabilidad_grupo06.json): `docs/grafana/observabilidad_grupo06.json`

### Orchestration

- [Pipeline DAG](https://github.com/fmiles-2006/ingdatos-clima-sidney-grupo06/blob/main/dags/pipeline_grupo06.py): `dags/pipeline_grupo06.py`

### Other components

- **Open-Meteo**

## Connections

- Pipeline DAG → Weather ingestion: runs (evidence: `dags/pipeline_grupo06.py`)
- Open-Meteo → Weather ingestion: provides weather (evidence: `README.md`)
- Weather ingestion → Delta bronze: writes Delta (evidence: `README.md`)
- Pipeline DAG → dbt project: runs dbt (evidence: `dags/pipeline_grupo06.py`)
- Delta bronze → Daily climate staging: feeds staging (evidence: `README.md`)
- Daily climate staging → Monthly climate facts: builds marts (evidence: `README.md`)
- Daily climate staging → Daily climate features: builds marts (evidence: `README.md`)
- Pipeline DAG → dbt quality checks: runs tests (evidence: `dags/pipeline_grupo06.py`)
- dbt project → DuckDB warehouse: builds models (evidence: `README.md`)
- Pipeline DAG → Gold publisher: runs publication (evidence: `dags/pipeline_grupo06.py`)
- Gold publisher → DuckDB warehouse: reads gold (evidence: `scripts/publicar_gold.py`)
- Gold publisher → PostgreSQL gold: publishes tables (evidence: `scripts/publicar_gold.py`)
- PostgreSQL gold → Metabase dashboard: serves dashboard (evidence: `README.md`)
- Grafana monitoring → Pipeline DAG: monitors runs
