# johnatanogueira/brazilian-public-data-lakehouse architecture

> Architecture diagram of the GitHub repository johnatanogueira/brazilian-public-data-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/johnatanogueira/brazilian-public-data-lakehouse
- Repository: https://github.com/johnatanogueira/brazilian-public-data-lakehouse

## Overview

This repository implements a local IBGE municipalities data lakehouse. A user triggers the Airflow DAG, which extracts API data into Bronze JSON, runs Spark processing for Silver Parquet and Gold analytical aggregates, then validates the outputs; failures stop the workflow. Logging is a shared internal layer. The README documents the Gold datasets and capabilities, but the sampled `ibge_gold.py` content appears to validate data rather than build aggregates, so that implementation detail is uncertain. Data paths are runtime outputs, not entries in the supplied tree.

Main entry points: [`dags/ibge_pipeline.py`](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/dags/ibge_pipeline.py) (IBGE pipeline DAG), [`src/transformation/ibge_gold.py`](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/transformation/ibge_gold.py) (Gold aggregates), [`src/quality/validate_gold.py`](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/quality/validate_gold.py) (Data quality checks), [`src/ingestion/ibge_api.py`](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/ingestion/ibge_api.py) (IBGE extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_dag["IBGE pipeline DAG<br/>[ibge_pipeline.py]"]
end

subgraph group_data_pipeline["Data pipeline"]
  node_extract["IBGE extraction<br/>[ibge_api.py]"]
  node_bronze[("Bronze JSON")]
  node_silver_transform["Silver transformation<br/>[ibge_silver.py]"]
  node_silver[("Silver Parquet")]
  node_gold_transform["Gold aggregates<br/>[ibge_gold.py]"]
  node_gold_state[("Municipalities by state")]
  node_gold_region[("Municipalities by region")]
  node_quality["Data quality checks<br/>[validate_gold.py]"]
end

subgraph group_shared["Shared services"]
  node_logger["Centralized logging<br/>[logger.py]"]
end

node_user(("Pipeline operator"))
node_ibge_api["IBGE municipalities API"]

node_user -->|"triggers"| node_dag
node_dag -->|"runs extraction"| node_extract
node_extract -->|"requests data"| node_ibge_api
node_extract -->|"writes JSON"| node_bronze
node_dag -->|"runs Spark"| node_silver_transform
node_silver_transform -->|"reads JSON"| node_bronze
node_silver_transform -->|"writes Parquet"| node_silver
node_dag -->|"runs Spark"| node_gold_transform
node_gold_transform -->|"reads Parquet"| node_silver
node_gold_transform -->|"produces aggregate"| node_gold_state
node_gold_transform -->|"produces aggregate"| node_gold_region
node_dag -->|"runs validation"| node_quality
node_quality -->|"reads Parquet"| node_silver
node_quality -->|"validates aggregate"| node_gold_state
node_quality -->|"validates aggregate"| node_gold_region
node_extract -->|"logs events"| node_logger
node_silver_transform -->|"logs events"| node_logger
node_gold_transform -->|"logs events"| node_logger
node_quality -->|"logs events"| node_logger

click node_dag "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/dags/ibge_pipeline.py"
click node_extract "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/ingestion/ibge_api.py"
click node_silver_transform "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/transformation/ibge_silver.py"
click node_gold_transform "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/transformation/ibge_gold.py"
click node_quality "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/quality/validate_gold.py"
click node_logger "https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/utils/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag toneBlue
class node_extract,node_bronze,node_silver_transform,node_silver,node_gold_transform,node_gold_state,node_gold_region,node_quality toneAmber
class node_logger,node_ibge_api toneMint
class node_user toneIndigo
```

## Components

### Orchestration

- [IBGE pipeline DAG](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/dags/ibge_pipeline.py): `dags/ibge_pipeline.py`

### Data pipeline

- [IBGE extraction](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/ingestion/ibge_api.py): `src/ingestion/ibge_api.py`
- **Bronze JSON**
- [Silver transformation](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/transformation/ibge_silver.py): `src/transformation/ibge_silver.py`
- **Silver Parquet**
- [Gold aggregates](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/transformation/ibge_gold.py): `src/transformation/ibge_gold.py`
- **Municipalities by state**
- **Municipalities by region**
- [Data quality checks](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/quality/validate_gold.py): `src/quality/validate_gold.py`

### Shared services

- [Centralized logging](https://github.com/johnatanogueira/brazilian-public-data-lakehouse/blob/main/src/utils/logger.py): `src/utils/logger.py`

### Other components

- **Pipeline operator**
- **IBGE municipalities API**

## Connections

- Pipeline operator → IBGE pipeline DAG: triggers
- IBGE pipeline DAG → IBGE extraction: runs extraction
- IBGE extraction → IBGE municipalities API: requests data
- IBGE extraction → Bronze JSON: writes JSON
- IBGE pipeline DAG → Silver transformation: runs Spark
- Silver transformation → Bronze JSON: reads JSON
- Silver transformation → Silver Parquet: writes Parquet
- IBGE pipeline DAG → Gold aggregates: runs Spark
- Gold aggregates → Silver Parquet: reads Parquet
- Gold aggregates → Municipalities by state: produces aggregate
- Gold aggregates → Municipalities by region: produces aggregate
- IBGE pipeline DAG → Data quality checks: runs validation
- Data quality checks → Silver Parquet: reads Parquet
- Data quality checks → Municipalities by state: validates aggregate
- Data quality checks → Municipalities by region: validates aggregate
- IBGE extraction → Centralized logging: logs events
- Silver transformation → Centralized logging: logs events
- Gold aggregates → Centralized logging: logs events
- Data quality checks → Centralized logging: logs events