How vitoraraujoooo/data-application-cidades works
Gov Hub BR integrates Brazilian public-sector data for analysis and public-management decisions. Its principal flow is scheduled source ingestion, raw and staging lake storage, dbt transformation into analytical data, then dashboard exploration; a separate historical-data treatment tool classifies and cleans legacy tables. Governance metadata synchronization is another product capability. The README documents the Medallion flow and Airflow/dbt/Superset stack; sampled DAGs and plugin references support the specific ingestion links shown. Other unsampled ingestion DAGs are represented by their existing directory, so their individual source-to-client wiring is not asserted.
Main entry points: bacen_sgs_ingest_dag.py BACEN ingestion · cliente_postgres.py PostgreSQL client · cliente_minio.py MinIO client · main.py Historical treatment runner
Components 21
Source ingestion
- Airflow ingestion DAGs
dags/data_ingest - IBGE ingestion
dags/data_ingest/ibge/ibge_ingest_dag.py - IBGE client
plugins/cliente_ibge.py - BACEN ingestion
dags/data_ingest/bacen/bacen_sgs_ingest_dag.py - BACEN client
plugins/cliente_bacen.py
Lake and storage
- PostgreSQL client
plugins/cliente_postgres.py - PostgreSQL
- MinIO client
plugins/cliente_minio.py - Raw and staging lake
- Staging conversion
plugins/ingestor_lake.py
Analytics and dashboards
- dbt transformations
dbt/mcid - dbt orchestration
dags/dbt/mcid_cosmos_dag.py - Dashboard workflows
dags/dashboards/dashboard_servidores_dag.py - Superset exploration
superset
Historical treatment
- Historical treatment runner
data-science/dados-historicos-tratamento/main.py - Classification and cleaning
data-science/dados-historicos-tratamento/src/classificacao/tratamento.py - Treatment database access
data-science/dados-historicos-tratamento/src/classificacao/db/reader.py
Metadata governance
- Metadata ingestion workflow
dags/openmetadata_ingestion_dag.py - OpenMetadata helpers
helpers/openmetadata/runner.py - OpenMetadata catalog
Other components
- Public data sources
Connections 25
- Public data sources → IBGE ingestion: provide data
README.md - IBGE ingestion → IBGE client: calls
dags/data_ingest/ibge/ibge_ingest_dag.py - IBGE client → Public data sources: fetches from
- IBGE ingestion → PostgreSQL client: stores records
dags/data_ingest/ibge/ibge_ingest_dag.py - PostgreSQL client → PostgreSQL: writes
plugins/cliente_postgres.py - Public data sources → BACEN ingestion: provide data
README.md - BACEN ingestion → BACEN client: calls
dags/data_ingest/bacen/bacen_sgs_ingest_dag.py - BACEN client → Public data sources: fetches from
- BACEN ingestion → PostgreSQL client: stores records
dags/data_ingest/bacen/bacen_sgs_ingest_dag.py - BACEN ingestion → MinIO client: uploads raw
dags/data_ingest/bacen/bacen_sgs_ingest_dag.py - MinIO client → Raw and staging lake: writes objects
plugins/cliente_minio.py - Airflow ingestion DAGs → MinIO client: uses clients
dags/data_ingest/ibge/ibge_ingest_dag.py - Staging conversion → MinIO client: uploads staging
plugins/ingestor_lake.py - Staging conversion → Raw and staging lake: writes parquet
plugins/ingestor_lake.py - dbt orchestration → dbt transformations: orchestrates
README.md - dbt transformations → Raw and staging lake: reads staging
- dbt transformations → PostgreSQL: loads silver
- Dashboard workflows → PostgreSQL client: queries
dags/dashboards/dashboard_servidores_dag.py - Dashboard workflows → Superset exploration: supports dashboards
README.md - Superset exploration → PostgreSQL: explores data
README.md - Historical treatment runner → Classification and cleaning: runs treatment
data-science/dados-historicos-tratamento/main.py - Historical treatment runner → Treatment database access: reads tables
data-science/dados-historicos-tratamento/main.py - Historical treatment runner → Treatment database access: writes results
data-science/dados-historicos-tratamento/main.py - Metadata ingestion workflow → OpenMetadata helpers: runs recipes
dags/openmetadata_ingestion_dag.py - Metadata ingestion workflow → OpenMetadata catalog: syncs catalog
dags/openmetadata_ingestion_dag.py