How migue-08/research_paper_ms_econ works
Project Type and Purpose
• This is an empirical research/data-analysis pipeline—NOT a full-stack web app.
• Purpose: ingest national exam (Saber11) and school census data (DANE), clean and merge them, run regressions, produce visualizations, and publish results as a static site via Quarto.
File Structure
• Top-level notebooks (.ipynb) for each processing step (cleaning, merging, visualizations).
• A Quarto munging script (munging.qmd) and a Quarto config (_quarto.yml).
• A STATA .do file (regression_analysis.do) for the final regression step.
• docs/ folder containing a generated static site (HTML, assets).
• requirements.txt lists Python/R dependencies.
Main Components
A. External Data Sources
• ICFES Saber11 microdata (2019 & 2022)
• DANE scholar census (2019 & 2022)
B. Data-Ingestion & Cleaning Layer
• Jupyter notebook: Saber_11_Cleaning.ipynb
• Jupyter notebook: Acceso_tecnología_por_estudiante.ipynb
C. Merging & Transformation Layer
• merged.ipynb (combines cleaned Saber11 + census data)
• munging.qmd (Quarto pipeline for further munging & summary stats)
D. Analysis Layer
• regression_analysis.do (STATA do-file executing regression models)
E. Visualization Layer
• Grafico_distribucion_datos.ipynb
• Matricula_ed_* notebooks
• Tasa_deserción.ipynb, etc.
F. Presentation / Documentation Layer
• Quarto (_quarto.yml) generates docs/ site
• docs/index.html + assets (bootstrap, quarto scripts)
Relationships & Data Flow
• Cleaning Notebooks → Merged Notebook → munging.qmd
• munging.qmd → Summary tables & cleaned artifacts → regression_analysis.do
• regression_analysis.do → Regression outputs
• All notebooks & munging scripts → Visualization Notebooks → Figures
• Quarto pipeline → Static HTML site in docs/ incorporating figures & narrative
Architectural Patterns & Design Principles
• Sequential ETL pipeline: Extract (download) → Transform (clean & merge) → Load (analysis & viz)
• Tool-chain integration: Python Jupyter → Quarto → STATA → Quarto static site
• Clear separation between data processing and documentation/publishing