How johnatanogueira/brazilian-public-data-lakehouse works
This repository implements a local IBGE municipalities data lakehouse. A user triggers the Airflow DAG, which extracts API data into Bronze JSON, runs Spark processing for Silver Parquet and Gold analytical aggregates, then validates the outputs; failures stop the workflow. Logging is a shared internal layer. The README documents the Gold datasets and capabilities, but the sampled ibge_gold.py content appears to validate data rather than build aggregates, so that implementation detail is uncertain. Data paths are runtime outputs, not entries in the supplied tree.
Main entry points: ibge_pipeline.py IBGE pipeline DAG · ibge_gold.py Gold aggregates · validate_gold.py Data quality checks · ibge_api.py IBGE extraction
Components 12
Orchestration
- IBGE pipeline DAG
dags/ibge_pipeline.py
Data pipeline
- IBGE extraction
src/ingestion/ibge_api.py - Bronze JSON
- Silver transformation
src/transformation/ibge_silver.py - Silver Parquet
- Gold aggregates
src/transformation/ibge_gold.py - Municipalities by state
- Municipalities by region
- Data quality checks
src/quality/validate_gold.py
Shared services
- Centralized logging
src/utils/logger.py
Other components
- Pipeline operator
- IBGE municipalities API
Connections 19
- Pipeline operator → IBGE pipeline DAG: triggers
- IBGE pipeline DAG → IBGE extraction: runs extraction
- IBGE extraction → IBGE municipalities API: requests data
- IBGE extraction → Bronze JSON: writes JSON
- IBGE pipeline DAG → Silver transformation: runs Spark
- Silver transformation → Bronze JSON: reads JSON
- Silver transformation → Silver Parquet: writes Parquet
- IBGE pipeline DAG → Gold aggregates: runs Spark
- Gold aggregates → Silver Parquet: reads Parquet
- Gold aggregates → Municipalities by state: produces aggregate
- Gold aggregates → Municipalities by region: produces aggregate
- IBGE pipeline DAG → Data quality checks: runs validation
- Data quality checks → Silver Parquet: reads Parquet
- Data quality checks → Municipalities by state: validates aggregate
- Data quality checks → Municipalities by region: validates aggregate
- IBGE extraction → Centralized logging: logs events
- Silver transformation → Centralized logging: logs events
- Gold aggregates → Centralized logging: logs events
- Data quality checks → Centralized logging: logs events