# vitoraraujoooo/data-application-cidades architecture

> Architecture diagram of the GitHub repository vitoraraujoooo/data-application-cidades, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vitoraraujoooo/data-application-cidades
- Repository: https://github.com/vitoraraujoooo/data-application-cidades

## Overview

Gov Hub BR integrates Brazilian public-sector data for analysis and public-management decisions. Its principal flow is scheduled source ingestion, raw and staging lake storage, dbt transformation into analytical data, then dashboard exploration; a separate historical-data treatment tool classifies and cleans legacy tables. Governance metadata synchronization is another product capability. The README documents the Medallion flow and Airflow/dbt/Superset stack; sampled DAGs and plugin references support the specific ingestion links shown. Other unsampled ingestion DAGs are represented by their existing directory, so their individual source-to-client wiring is not asserted.

Main entry points: [`dags/data_ingest/bacen/bacen_sgs_ingest_dag.py`](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/data_ingest/bacen/bacen_sgs_ingest_dag.py) (BACEN ingestion), [`plugins/cliente_postgres.py`](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_postgres.py) (PostgreSQL client), [`plugins/cliente_minio.py`](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_minio.py) (MinIO client), [`data-science/dados-historicos-tratamento/main.py`](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/main.py) (Historical treatment runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_airflow["Airflow ingestion DAGs"]
  node_ibge_dag["IBGE ingestion<br/>[ibge_ingest_dag.py]"]
  node_ibge_client["IBGE client<br/>[cliente_ibge.py]"]
  node_bacen_dag["BACEN ingestion"]
  node_bacen_client["BACEN client<br/>[cliente_bacen.py]"]
end

subgraph group_lake["Lake and storage"]
  node_postgres_client["PostgreSQL client"]
  node_postgres[("PostgreSQL")]
  node_minio_client["MinIO client<br/>[cliente_minio.py]"]
  node_minio[("Raw and staging lake")]
  node_lake_transform["Staging conversion<br/>[ingestor_lake.py]"]
end

subgraph group_analytics["Analytics and dashboards"]
  node_dbt["dbt transformations"]
  node_dbt_dag["dbt orchestration<br/>[mcid_cosmos_dag.py]"]
  node_dashboards["Dashboard workflows"]
  node_superset["Superset exploration"]
end

subgraph group_historical["Historical treatment"]
  node_treatment_entry["Historical treatment runner<br/>[main.py]"]
  node_classification["Classification and cleaning<br/>[tratamento.py]"]
  node_historical_db["Treatment database access<br/>[reader.py]"]
end

subgraph group_governance["Metadata governance"]
  node_metadata_dag["Metadata ingestion workflow"]
  node_metadata_helpers["OpenMetadata helpers<br/>[runner.py]"]
  node_openmetadata["OpenMetadata catalog"]
end

node_agencies(("Public data sources"))

node_agencies -->|"provide data"| node_ibge_dag
node_ibge_dag -->|"calls"| node_ibge_client
node_ibge_client -.->|"fetches from"| node_agencies
node_ibge_dag -->|"stores records"| node_postgres_client
node_postgres_client -->|"writes"| node_postgres
node_agencies -->|"provide data"| node_bacen_dag
node_bacen_dag -->|"calls"| node_bacen_client
node_bacen_client -.->|"fetches from"| node_agencies
node_bacen_dag -->|"stores records"| node_postgres_client
node_bacen_dag -->|"uploads raw"| node_minio_client
node_minio_client -->|"writes objects"| node_minio
node_airflow -.->|"uses clients"| node_minio_client
node_lake_transform -->|"uploads staging"| node_minio_client
node_lake_transform -->|"writes parquet"| node_minio
node_dbt_dag -->|"orchestrates"| node_dbt
node_dbt -->|"reads staging"| node_minio
node_dbt -->|"loads silver"| node_postgres
node_dashboards -->|"queries"| node_postgres_client
node_dashboards -->|"supports dashboards"| node_superset
node_superset -->|"explores data"| node_postgres
node_treatment_entry -->|"runs treatment"| node_classification
node_treatment_entry -->|"reads tables"| node_historical_db
node_treatment_entry -->|"writes results"| node_historical_db
node_metadata_dag -->|"runs recipes"| node_metadata_helpers
node_metadata_dag -->|"syncs catalog"| node_openmetadata

click node_airflow "https://github.com/vitoraraujoooo/data-application-cidades/tree/main/dags/data_ingest"
click node_ibge_dag "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/data_ingest/ibge/ibge_ingest_dag.py"
click node_ibge_client "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_ibge.py"
click node_bacen_dag "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/data_ingest/bacen/bacen_sgs_ingest_dag.py"
click node_bacen_client "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_bacen.py"
click node_postgres_client "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_postgres.py"
click node_minio_client "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_minio.py"
click node_lake_transform "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/ingestor_lake.py"
click node_dbt "https://github.com/vitoraraujoooo/data-application-cidades/tree/main/dbt/mcid"
click node_dbt_dag "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/dbt/mcid_cosmos_dag.py"
click node_dashboards "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/dashboards/dashboard_servidores_dag.py"
click node_superset "https://github.com/vitoraraujoooo/data-application-cidades/tree/main/superset"
click node_treatment_entry "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/main.py"
click node_classification "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/src/classificacao/tratamento.py"
click node_historical_db "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/src/classificacao/db/reader.py"
click node_metadata_dag "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/openmetadata_ingestion_dag.py"
click node_metadata_helpers "https://github.com/vitoraraujoooo/data-application-cidades/blob/main/helpers/openmetadata/runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow,node_ibge_dag,node_ibge_client,node_bacen_dag,node_bacen_client toneBlue
class node_postgres_client,node_postgres,node_minio_client,node_minio,node_lake_transform toneAmber
class node_dbt,node_dbt_dag,node_dashboards,node_superset toneMint
class node_treatment_entry,node_classification,node_historical_db toneRose
class node_metadata_dag,node_metadata_helpers,node_openmetadata,node_agencies toneIndigo
```

## Components

### Source ingestion

- [Airflow ingestion DAGs](https://github.com/vitoraraujoooo/data-application-cidades/tree/main/dags/data_ingest): `dags/data_ingest`
- [IBGE ingestion](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/data_ingest/ibge/ibge_ingest_dag.py): `dags/data_ingest/ibge/ibge_ingest_dag.py`
- [IBGE client](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_ibge.py): `plugins/cliente_ibge.py`
- [BACEN ingestion](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/data_ingest/bacen/bacen_sgs_ingest_dag.py): `dags/data_ingest/bacen/bacen_sgs_ingest_dag.py`
- [BACEN client](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_bacen.py): `plugins/cliente_bacen.py`

### Lake and storage

- [PostgreSQL client](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_postgres.py): `plugins/cliente_postgres.py`
- **PostgreSQL**
- [MinIO client](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/cliente_minio.py): `plugins/cliente_minio.py`
- **Raw and staging lake**
- [Staging conversion](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/plugins/ingestor_lake.py): `plugins/ingestor_lake.py`

### Analytics and dashboards

- [dbt transformations](https://github.com/vitoraraujoooo/data-application-cidades/tree/main/dbt/mcid): `dbt/mcid`
- [dbt orchestration](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/dbt/mcid_cosmos_dag.py): `dags/dbt/mcid_cosmos_dag.py`
- [Dashboard workflows](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/dashboards/dashboard_servidores_dag.py): `dags/dashboards/dashboard_servidores_dag.py`
- [Superset exploration](https://github.com/vitoraraujoooo/data-application-cidades/tree/main/superset): `superset`

### Historical treatment

- [Historical treatment runner](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/main.py): `data-science/dados-historicos-tratamento/main.py`
- [Classification and cleaning](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/src/classificacao/tratamento.py): `data-science/dados-historicos-tratamento/src/classificacao/tratamento.py`
- [Treatment database access](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/data-science/dados-historicos-tratamento/src/classificacao/db/reader.py): `data-science/dados-historicos-tratamento/src/classificacao/db/reader.py`

### Metadata governance

- [Metadata ingestion workflow](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/dags/openmetadata_ingestion_dag.py): `dags/openmetadata_ingestion_dag.py`
- [OpenMetadata helpers](https://github.com/vitoraraujoooo/data-application-cidades/blob/main/helpers/openmetadata/runner.py): `helpers/openmetadata/runner.py`
- **OpenMetadata catalog**

### Other components

- **Public data sources**

## Connections

- Public data sources → IBGE ingestion: provide data (evidence: `README.md`)
- IBGE ingestion → IBGE client: calls (evidence: `dags/data_ingest/ibge/ibge_ingest_dag.py`)
- IBGE client → Public data sources: fetches from
- IBGE ingestion → PostgreSQL client: stores records (evidence: `dags/data_ingest/ibge/ibge_ingest_dag.py`)
- PostgreSQL client → PostgreSQL: writes (evidence: `plugins/cliente_postgres.py`)
- Public data sources → BACEN ingestion: provide data (evidence: `README.md`)
- BACEN ingestion → BACEN client: calls (evidence: `dags/data_ingest/bacen/bacen_sgs_ingest_dag.py`)
- BACEN client → Public data sources: fetches from
- BACEN ingestion → PostgreSQL client: stores records (evidence: `dags/data_ingest/bacen/bacen_sgs_ingest_dag.py`)
- BACEN ingestion → MinIO client: uploads raw (evidence: `dags/data_ingest/bacen/bacen_sgs_ingest_dag.py`)
- MinIO client → Raw and staging lake: writes objects (evidence: `plugins/cliente_minio.py`)
- Airflow ingestion DAGs → MinIO client: uses clients (evidence: `dags/data_ingest/ibge/ibge_ingest_dag.py`)
- Staging conversion → MinIO client: uploads staging (evidence: `plugins/ingestor_lake.py`)
- Staging conversion → Raw and staging lake: writes parquet (evidence: `plugins/ingestor_lake.py`)
- dbt orchestration → dbt transformations: orchestrates (evidence: `README.md`)
- dbt transformations → Raw and staging lake: reads staging
- dbt transformations → PostgreSQL: loads silver
- Dashboard workflows → PostgreSQL client: queries (evidence: `dags/dashboards/dashboard_servidores_dag.py`)
- Dashboard workflows → Superset exploration: supports dashboards (evidence: `README.md`)
- Superset exploration → PostgreSQL: explores data (evidence: `README.md`)
- Historical treatment runner → Classification and cleaning: runs treatment (evidence: `data-science/dados-historicos-tratamento/main.py`)
- Historical treatment runner → Treatment database access: reads tables (evidence: `data-science/dados-historicos-tratamento/main.py`)
- Historical treatment runner → Treatment database access: writes results (evidence: `data-science/dados-historicos-tratamento/main.py`)
- Metadata ingestion workflow → OpenMetadata helpers: runs recipes (evidence: `dags/openmetadata_ingestion_dag.py`)
- Metadata ingestion workflow → OpenMetadata catalog: syncs catalog (evidence: `dags/openmetadata_ingestion_dag.py`)