leonardo-michelotti/observatorio-educacao-rs

Purpose and entry points

This is an end-to-end local data product analyzing basic-education indicators for Santa Maria, Rio Grande do Sul, and Brazil. Its primary execution entry point is run_pipeline.py, which runs extraction, dbt transformation, chart creation, and static-page generation. Published entry points are public/index.html, public/arquitetura.html, and machine-readable freshness metadata in public/data-status.json.

Source ingestion and bronze boundary

Two ingestion paths deliberately isolate external data acquisition from analytical logic:

• Official INEP XLS/XLSX downloads for approval rates and age-grade distortion: ingestion/extract_inep.py, with source definitions and validation references in ingestion/inep_sources.json and ingestion/inep_reference_values.json.

• BigQuery/Base dos Dados extraction for IDEB and SAEB: ingestion/extract_bd.py.

• A snapshot restoration path supports refreshes without historical BigQuery queries: ingestion/load_ideb_snapshot.py.

Bronze outputs are Parquet plus provenance, separating raw source evidence from normalized analytics.

Transformation and analytical contract

DuckDB is the local analytical engine, with dbt implementing the silver/gold layers:

• Staging normalizes source-specific structures: dbt/models/staging/stg_ideb.sql and dbt/models/staging/stg_indicadores.sql.

• The mart exposes one tidy fact contract: dbt/models/marts/fct_indicadores.sql.

• The intended grain is indicator, geographic level, education stage, year, and value. Schema and semantic constraints are defined in dbt/models/marts/_marts.yml, with grain/range checks in dbt/tests/fct_indicadores_unique_grain.sql and dbt/tests/fct_indicadores_valid_ranges.sql.

This fact model is the sole intended boundary for downstream visualization and future indicators.

Product-generation layer

Visualization reads transformed DuckDB data rather than source files:

• viz/make_charts.py produces versioned PNG assets used by the README.

• viz/build_dashboard.py produces self-contained editorial analysis and architecture pages.

• Generated web artifacts are copied to public/index.html and public/arquitetura.html; working visual templates/artifacts live in viz/dashboard.html and viz/arquitetura.html.

The dashboard includes interactive charts, table alternatives, responsive behavior, and data-status publication.

Orchestration and update control

run_pipeline.py is the central control plane for a full reproducible build. It sequences ingestion, dbt build, chart generation, and page generation. Offline CI uses deterministic synthetic bronze data, while the manual official refresh workflow restores audited history and upserts the newest INEP year. This distinction prevents test fixtures from entering published outputs.

Runtime, infrastructure, and deployment

The main runtime is Python 3.12+ with dbt-duckdb, DuckDB, matplotlib, and BigQuery client access. BigQuery requires Google Cloud ADC and a billing project for IDEB/SAEB extraction. Static deployment uses Caddy in a container:

• Dockerfile

• Caddyfile

• railway.toml

Railway serves the generated static public/ directory over HTTPS; raw data and credentials are excluded from the deploy image.

Automation and quality gates

GitHub Actions separates responsibilities:

• .github/workflows/ci.yml rebuilds offline, runs dbt and Python tests, and validates desktop/mobile navigation and WCAG behavior through Playwright.

• .github/workflows/refresh-inep.yml performs real official-source refreshes, preserves provenance/artifacts, and opens update PRs.

• .github/workflows/deploy.yml deploys validated main-branch output to Railway.