How gianse/projeto-caged works
This repository builds and analyzes Brazilian CAGED and RAIS employment datasets: it extracts source files, produces bronze and silver parquet layers, derives gold datasets, and serves analysis through dashboards. The main user-facing workflow is viewing labor-market indicators; operators also have a separate extraction-observability panel and can trigger jobs there. The tree identifies additional forecasting, nowcasting, survival, clustering, and wage-gap analyses. Source excerpts are partial and the README is empty, so uncertain implementation links are omitted.
Main entry points: silver_caged/construir_silver.py CAGED silver · construir.py Unified gold · run_extracao.py FTP extraction · silver_rais/construir_silver.py RAIS silver
Components 24
Ingestion
- FTP extraction
tasks_python/extracao_ftp/run_extracao.py - Bronze transform
tasks_python/extracao_ftp/transformador.py - Bronze parquet
tasks_python/publicar_bronze.py
Data curation
- CAGED silver
tasks_python/silver_caged/construir_silver.py - RAIS silver
tasks_python/silver_rais/construir_silver.py - Technology scope
tasks_python/gold_caged/escopo_tecnologia.py - Data audits
tasks_python/auditoria - Silver parquet
tasks_python/silver_caged/publicar_hf.py
Data products
- CAGED gold
tasks_python/gold_caged/construir_gold.py - RAIS gold
tasks_python/gold_rais/construir_gold.py - Unified gold
tasks_python/gold_unificado/construir.py - Gold datasets
tasks_python/gold_unificado/publicar_gold.py
Analysis
- Labor analyses
tasks_python/ciencia_dados - Balance forecast
tasks_python/ciencia_dados/previsao_saldo.py - Stock nowcast
tasks_python/ciencia_dados/nowcast_estoque.py - Wage-gap study
tasks_python/ciencia_dados/hiato_salarial.py
Interfaces
- Labor dashboard
tasks_python/dashboard/app.py - Pipeline monitor
painel/app.py - Job control
painel/processos.py
Other components
- MTE source
- Operator
- Dashboard viewer
- Hugging Face
- MinIO storage
Connections 33
- MTE source → FTP extraction: provides files
- Operator → FTP extraction: starts extraction
- FTP extraction → Bronze transform: hands off files
- Bronze transform → MinIO storage: writes bronze
- MinIO storage → Bronze parquet: reads bronze
- Bronze parquet → Hugging Face: publishes datasets
- MinIO storage → CAGED silver: reads bronze
- MinIO storage → RAIS silver: reads bronze
- Technology scope → CAGED silver: defines filter
- CAGED silver → MinIO storage: writes silver
- RAIS silver → MinIO storage: writes silver
- Data audits → MinIO storage: checks and adjusts
- CAGED silver → Hugging Face: publishes silver
- RAIS silver → Hugging Face: publishes silver
- MinIO storage → CAGED gold: reads silver
- MinIO storage → RAIS gold: reads silver
- CAGED gold → MinIO storage: writes gold
- RAIS gold → MinIO storage: writes gold
- CAGED gold → Unified gold: supplies product data
- RAIS gold → Unified gold: supplies product data
- Unified gold → MinIO storage: writes gold
- Unified gold → Hugging Face: publishes gold
- MinIO storage → Labor analyses: provides datasets
- Labor analyses → MinIO storage: materializes analyses
- MinIO storage → Balance forecast: reads series
- MinIO storage → Stock nowcast: reads datasets
- MinIO storage → Wage-gap study: reads RAIS data
- Dashboard viewer → Labor dashboard: views indicators
- Labor dashboard → Hugging Face: reads published data
- Operator → Pipeline monitor: monitors jobs
- Pipeline monitor → MinIO storage: reads storage status
- Pipeline monitor → Job control: starts or stops
- Job control → FTP extraction: triggers extraction