How gabriel-goncalves1122/mapeamento-risco-queimadas-inpe works
CCO DataOps (SIRDA) monitors wildfire impacts on critical infrastructure through a Bronze–Silver–Gold data platform. Its workflow acquires official data, builds cleaned spatial dimensions, enriches and clusters fire observations, then publishes fact streams for OLAP analysis and geospatial dashboards. Dagster coordinates documented pipeline assets; the repository also includes Druid lookup registration and network-centrality analysis. The README describes broader integrations and stages than the excerpts expose, so those links are cited to the README rather than inferred from unsampled code.
Main entry points: repo.py Dagster assets · processador.py Focus processor · etl Dimension ETLs · produtor_eventos.py Event producer
Components 23
Core and acquisition
- Pipeline contracts
src/core/contracts/contracts.py - Pipeline settings
src/core/config/settings.py - ANEEL extractor
src/bronze/extractors/ingestao_aneel.py - INPE extractor
src/bronze/extractors/inpe_queimadas_extractor.py
Silver processing
- Dimension ETLs
src/silver/etl - Hospital ETL
src/silver/etl/etl_dim_hospitais.py - Silver utilities
src/silver/utils - Spatial engine
src/silver/spatial - Focus processor
src/silver/streaming_processing/processador.py - Focus calculators
src/silver/streaming_processing/processors/calculators.py - Druid lookups
src/silver/lookups/druid_lookup_registrar.py
Gold analytics
- Fire event clustering
src/gold/science/dbscan/eventizador.py - Event geometry
src/gold/science/dbscan/event_geometry_builder.py - Grid centrality
src/gold/science/networkx/network_builder.py - Focus producer
src/gold/streaming/produtor_focos.py - Event producer
src/gold/streaming/produtor_eventos.py - Kafka client
src/gold/streaming/kafka_client.py - Streaming audit
src/gold/streaming/streaming_audit_logger.py
Orchestration and OLAP
- Dagster assets
src/orchestration/dagster/repo.py - Druid OLAP
- Superset dashboards
Other components
- Official data sources
- Kafka broker
Connections 28
- Official data sources → ANEEL extractor: provides data
src/bronze/extractors/ingestao_aneel.py - Official data sources → INPE extractor: provides data
README.md - ANEEL extractor → Dimension ETLs: supplies raw layers
README.md - Dimension ETLs → Silver utilities: uses helpers
src/silver/etl/etl_dim_hospitais.py - Hospital ETL → Silver utilities: uses helpers
src/silver/etl/etl_dim_hospitais.py - Hospital ETL → Dimension ETLs: builds dimension
src/silver/etl/etl_dim_hospitais.py - Focus processor → Focus calculators: calls calculations
src/silver/streaming_processing/processador.py - Focus processor → Spatial engine: queries threats
src/silver/streaming_processing/processador.py - Focus processor → Silver utilities: cleans values
src/silver/streaming_processing/processador.py - Fire event clustering → Event geometry: builds geometry
src/gold/science/dbscan/eventizador.py - Fire event clustering → Focus processor: reuses enrichment
- Grid centrality → Dimension ETLs: analyzes grid
- Focus producer → Kafka client: publishes through
- Event producer → Fire event clustering: clusters observations
src/gold/streaming/produtor_eventos.py - Event producer → Focus processor: processes observations
src/gold/streaming/produtor_eventos.py - Event producer → Kafka client: publishes through
src/gold/streaming/produtor_eventos.py - Event producer → Streaming audit: records activity
src/gold/streaming/produtor_eventos.py - Kafka client → Kafka broker: sends messages
src/gold/streaming/kafka_client.py - Kafka broker → Druid OLAP: feeds OLAP
README.md - Druid lookups → Druid OLAP: registers lookups
README.md - Druid OLAP → Superset dashboards: supplies analytics
README.md - Dagster assets → ANEEL extractor: executes asset
src/orchestration/dagster/repo.py - Dagster assets → INPE extractor: executes asset
src/orchestration/dagster/repo.py - Dagster assets → Dimension ETLs: orchestrates assets
src/orchestration/dagster/repo.py - Dagster assets → Grid centrality: executes asset
src/orchestration/dagster/repo.py - Dagster assets → Fire event clustering: executes asset
src/orchestration/dagster/repo.py - Pipeline settings → Kafka client: configures client
src/gold/streaming/kafka_client.py - Pipeline contracts → Dagster assets: defines step contract
src/orchestration/dagster/repo.py