# gianse/projeto-caged architecture

> Architecture diagram of the GitHub repository gianse/projeto-caged, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gianse/projeto-caged
- Repository: https://github.com/gianse/projeto-caged

## Overview

This repository builds and analyzes Brazilian CAGED and RAIS employment datasets: it extracts source files, produces bronze and silver parquet layers, derives gold datasets, and serves analysis through dashboards. The main user-facing workflow is viewing labor-market indicators; operators also have a separate extraction-observability panel and can trigger jobs there. The tree identifies additional forecasting, nowcasting, survival, clustering, and wage-gap analyses. Source excerpts are partial and the README is empty, so uncertain implementation links are omitted.

Main entry points: [`tasks_python/silver_caged/construir_silver.py`](https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_caged/construir_silver.py) (CAGED silver), [`tasks_python/gold_unificado/construir.py`](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_unificado/construir.py) (Unified gold), [`tasks_python/extracao_ftp/run_extracao.py`](https://github.com/gianse/projeto-caged/blob/main/tasks_python/extracao_ftp/run_extracao.py) (FTP extraction), [`tasks_python/silver_rais/construir_silver.py`](https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_rais/construir_silver.py) (RAIS silver)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_ftp["FTP extraction<br/>[run_extracao.py]"]
  node_transform["Bronze transform<br/>[transformador.py]"]
  node_bronze[("Bronze parquet<br/>[publicar_bronze.py]")]
end

subgraph group_curation["Data curation"]
  node_silverc["CAGED silver"]
  node_silverr["RAIS silver"]
  node_scope["Technology scope"]
  node_audit["Data audits"]
  node_silverstore[("Silver parquet<br/>[publicar_hf.py]")]
end

subgraph group_products["Data products"]
  node_goldc["CAGED gold<br/>[construir_gold.py]"]
  node_goldr["RAIS gold<br/>[construir_gold.py]"]
  node_unified["Unified gold<br/>[construir.py]"]
  node_goldstore[("Gold datasets<br/>[publicar_gold.py]")]
end

subgraph group_analysis["Analysis"]
  node_research["Labor analyses"]
  node_forecast["Balance forecast<br/>[previsao_saldo.py]"]
  node_nowcast["Stock nowcast<br/>[nowcast_estoque.py]"]
  node_wagegap["Wage-gap study<br/>[hiato_salarial.py]"]
end

subgraph group_interfaces["Interfaces"]
  node_dashboard["Labor dashboard<br/>[app.py]"]
  node_observability["Pipeline monitor<br/>[app.py]"]
  node_processes["Job control<br/>[processos.py]"]
end

node_mte(("MTE source"))
node_operator(("Operator"))
node_viewer(("Dashboard viewer"))
node_hf[("Hugging Face")]
node_minio[("MinIO storage")]

node_mte -->|"provides files"| node_ftp
node_operator -->|"starts extraction"| node_ftp
node_ftp -->|"hands off files"| node_transform
node_transform -->|"writes bronze"| node_minio
node_minio -->|"reads bronze"| node_bronze
node_bronze -->|"publishes datasets"| node_hf
node_minio -->|"reads bronze"| node_silverc
node_minio -->|"reads bronze"| node_silverr
node_scope -->|"defines filter"| node_silverc
node_silverc -->|"writes silver"| node_minio
node_silverr -->|"writes silver"| node_minio
node_audit -->|"checks and adjusts"| node_minio
node_silverc -->|"publishes silver"| node_hf
node_silverr -->|"publishes silver"| node_hf
node_minio -->|"reads silver"| node_goldc
node_minio -->|"reads silver"| node_goldr
node_goldc -->|"writes gold"| node_minio
node_goldr -->|"writes gold"| node_minio
node_goldc -->|"supplies product data"| node_unified
node_goldr -->|"supplies product data"| node_unified
node_unified -->|"writes gold"| node_minio
node_unified -->|"publishes gold"| node_hf
node_minio -->|"provides datasets"| node_research
node_research -->|"materializes analyses"| node_minio
node_minio -->|"reads series"| node_forecast
node_minio -->|"reads datasets"| node_nowcast
node_minio -->|"reads RAIS data"| node_wagegap
node_viewer -->|"views indicators"| node_dashboard
node_dashboard -->|"reads published data"| node_hf
node_operator -->|"monitors jobs"| node_observability
node_observability -->|"reads storage status"| node_minio
node_observability -->|"starts or stops"| node_processes
node_processes -->|"triggers extraction"| node_ftp

click node_ftp "https://github.com/gianse/projeto-caged/blob/main/tasks_python/extracao_ftp/run_extracao.py"
click node_transform "https://github.com/gianse/projeto-caged/blob/main/tasks_python/extracao_ftp/transformador.py"
click node_bronze "https://github.com/gianse/projeto-caged/blob/main/tasks_python/publicar_bronze.py"
click node_silverc "https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_caged/construir_silver.py"
click node_silverr "https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_rais/construir_silver.py"
click node_scope "https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_caged/escopo_tecnologia.py"
click node_audit "https://github.com/gianse/projeto-caged/tree/main/tasks_python/auditoria"
click node_silverstore "https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_caged/publicar_hf.py"
click node_goldc "https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_caged/construir_gold.py"
click node_goldr "https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_rais/construir_gold.py"
click node_unified "https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_unificado/construir.py"
click node_goldstore "https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_unificado/publicar_gold.py"
click node_research "https://github.com/gianse/projeto-caged/tree/main/tasks_python/ciencia_dados"
click node_forecast "https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/previsao_saldo.py"
click node_nowcast "https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/nowcast_estoque.py"
click node_wagegap "https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/hiato_salarial.py"
click node_dashboard "https://github.com/gianse/projeto-caged/blob/main/tasks_python/dashboard/app.py"
click node_observability "https://github.com/gianse/projeto-caged/blob/main/painel/app.py"
click node_processes "https://github.com/gianse/projeto-caged/blob/main/painel/processos.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ftp,node_transform,node_bronze,node_viewer toneBlue
class node_silverc,node_silverr,node_scope,node_audit,node_silverstore,node_hf,node_minio toneAmber
class node_goldc,node_goldr,node_unified,node_goldstore toneMint
class node_research,node_forecast,node_nowcast,node_wagegap toneRose
class node_dashboard,node_observability,node_processes,node_mte,node_operator toneIndigo
```

## Components

### Ingestion

- [FTP extraction](https://github.com/gianse/projeto-caged/blob/main/tasks_python/extracao_ftp/run_extracao.py): `tasks_python/extracao_ftp/run_extracao.py`
- [Bronze transform](https://github.com/gianse/projeto-caged/blob/main/tasks_python/extracao_ftp/transformador.py): `tasks_python/extracao_ftp/transformador.py`
- [Bronze parquet](https://github.com/gianse/projeto-caged/blob/main/tasks_python/publicar_bronze.py): `tasks_python/publicar_bronze.py`

### Data curation

- [CAGED silver](https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_caged/construir_silver.py): `tasks_python/silver_caged/construir_silver.py`
- [RAIS silver](https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_rais/construir_silver.py): `tasks_python/silver_rais/construir_silver.py`
- [Technology scope](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_caged/escopo_tecnologia.py): `tasks_python/gold_caged/escopo_tecnologia.py`
- [Data audits](https://github.com/gianse/projeto-caged/tree/main/tasks_python/auditoria): `tasks_python/auditoria`
- [Silver parquet](https://github.com/gianse/projeto-caged/blob/main/tasks_python/silver_caged/publicar_hf.py): `tasks_python/silver_caged/publicar_hf.py`

### Data products

- [CAGED gold](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_caged/construir_gold.py): `tasks_python/gold_caged/construir_gold.py`
- [RAIS gold](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_rais/construir_gold.py): `tasks_python/gold_rais/construir_gold.py`
- [Unified gold](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_unificado/construir.py): `tasks_python/gold_unificado/construir.py`
- [Gold datasets](https://github.com/gianse/projeto-caged/blob/main/tasks_python/gold_unificado/publicar_gold.py): `tasks_python/gold_unificado/publicar_gold.py`

### Analysis

- [Labor analyses](https://github.com/gianse/projeto-caged/tree/main/tasks_python/ciencia_dados): `tasks_python/ciencia_dados`
- [Balance forecast](https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/previsao_saldo.py): `tasks_python/ciencia_dados/previsao_saldo.py`
- [Stock nowcast](https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/nowcast_estoque.py): `tasks_python/ciencia_dados/nowcast_estoque.py`
- [Wage-gap study](https://github.com/gianse/projeto-caged/blob/main/tasks_python/ciencia_dados/hiato_salarial.py): `tasks_python/ciencia_dados/hiato_salarial.py`

### Interfaces

- [Labor dashboard](https://github.com/gianse/projeto-caged/blob/main/tasks_python/dashboard/app.py): `tasks_python/dashboard/app.py`
- [Pipeline monitor](https://github.com/gianse/projeto-caged/blob/main/painel/app.py): `painel/app.py`
- [Job control](https://github.com/gianse/projeto-caged/blob/main/painel/processos.py): `painel/processos.py`

### Other components

- **MTE source**
- **Operator**
- **Dashboard viewer**
- **Hugging Face**
- **MinIO storage**

## Connections

- MTE source → FTP extraction: provides files
- Operator → FTP extraction: starts extraction
- FTP extraction → Bronze transform: hands off files
- Bronze transform → MinIO storage: writes bronze
- MinIO storage → Bronze parquet: reads bronze
- Bronze parquet → Hugging Face: publishes datasets
- MinIO storage → CAGED silver: reads bronze
- MinIO storage → RAIS silver: reads bronze
- Technology scope → CAGED silver: defines filter
- CAGED silver → MinIO storage: writes silver
- RAIS silver → MinIO storage: writes silver
- Data audits → MinIO storage: checks and adjusts
- CAGED silver → Hugging Face: publishes silver
- RAIS silver → Hugging Face: publishes silver
- MinIO storage → CAGED gold: reads silver
- MinIO storage → RAIS gold: reads silver
- CAGED gold → MinIO storage: writes gold
- RAIS gold → MinIO storage: writes gold
- CAGED gold → Unified gold: supplies product data
- RAIS gold → Unified gold: supplies product data
- Unified gold → MinIO storage: writes gold
- Unified gold → Hugging Face: publishes gold
- MinIO storage → Labor analyses: provides datasets
- Labor analyses → MinIO storage: materializes analyses
- MinIO storage → Balance forecast: reads series
- MinIO storage → Stock nowcast: reads datasets
- MinIO storage → Wage-gap study: reads RAIS data
- Dashboard viewer → Labor dashboard: views indicators
- Labor dashboard → Hugging Face: reads published data
- Operator → Pipeline monitor: monitors jobs
- Pipeline monitor → MinIO storage: reads storage status
- Pipeline monitor → Job control: starts or stops
- Job control → FTP extraction: triggers extraction