How jvargasmu/population_estimation works
Project Type and Purpose
• This is an open-source, research-oriented Python codebase implementing POMELO: a fine-grained population mapping pipeline. It ingests coarse census counts and open geodata (rasters, building footprints), preprocesses them, trains a superpixel-based deep learning model (PyTorch + optional MRF), and outputs high-resolution population maps.
File Structure Analysis
• Top-level scripts: data preprocessing (e.g., preprocessing_pop_data.py, preprocessing_zmb_pop_data.py), feature computation (compute_gbuilding_count_map.py, compute_graph.py, compute_percentiles_from_popmap.py), baseline models (building_disagg_baseline.py), model training (superpixel_disagg_model.py, train_model_with_agg_data*.py, train_mrf.py), and visualization (show_feat_importance.py, visualization_*.py).
• Packages:
• cy_utils: Cython extensions for heavy computations (setup.py, .c/.pyx files).
• pix_transform: Python module for pixel-level transforms and evaluation.
• Config: config_pop.py holds all input paths, “no data” values, normalization stats.
• Requirements: requirements.txt specifies dependencies (numpy, torch, gdal, fiona, etc.).
Key README Insights
• The pipeline expects georeferenced raster covariates (100 m resolution) and census shapefiles.
• Steps: configure paths in config_pop.py → preprocess census into .pkl → train five cross-validation folds with superpixel_disagg_model.py → ensemble/final inference → generate .tif maps → optional Google Earth Engine export or demo interface.
• Uses WandB for experiment tracking, GDAL/Fiona for I/O, PyTorch for modeling.
Main Components
A. Data Sources (Inputs)
• Raster Covariates (WorldPop, night-lights, building counts)
• Census Data & Admin Boundaries (shapefiles, CSV)
B. Configuration
• config_pop.py (paths, norms, no-data values)
C. Preprocessing Module
• preprocessing_pop_data.py → outputs preprocessed census pickle
• compute_gbuilding_count_map.py, compute_graph.py, compute_percentiles_from_popmap.py → feature rasters/arrays
• convert_gbuilding_csv_to_shapefile.py → building footprints
• cy_utils (Cython routines) for compute-heavy steps
D. Pixel Transformation Module
• pix_transform package: transforms raw rasters into model features, evaluation metrics
E. Dataset Preparation
• Collate features + labels into dataset_dir for training
F. Training Module
• superpixel_disagg_model.py (five-fold cross-val & final ensemble)
• train_mrf.py (optional Markov Random Field model)
• train_model_with_agg_data*.py (alternative aggregated-data training)
• Underlying: PyTorch model, GPU acceleration
G. Experiment Tracking & Checkpoints
• WandB integration, checkpoints/ directory
H. Inference & Map Generation
• superpixel_disagg_model.py with “-e5f” flag → high-res .tif outputs
I. Visualization & Evaluation
• show_feat_importance.py, visualization_of_fine_pred_and_gt.py, visualization_of_ocr_pred_and_gt.py
Relationships and Data Flow
• Data Sources → Configuration → Preprocessing → Pixel Transformation → Dataset → Training → Inference → Visualization → Final Maps
• cy_utils and utils.py are cross-cutting libraries imported by preprocessing, transformation, and training scripts.
• Checkpoints feed into the final ensemble step.
• WandB and local logs capture metrics at training time.
Architectural Patterns and Principles
• Modular design: each script/package focuses on one responsibility.
• Cross-language extension: Cython (cy_utils) for performance-critical loops.
• Config-driven: config_pop.py centralizes paths and normalization metadata.
Technologies & Libraries
• Core: Python 3, NumPy, PyTorch, scikit-learn
• Geospatial I/O: GDAL (via gdal==3.2.1), Fiona, rasterio (implicit via GDAL), shapely
• Cython: compiled extensions in cy_utils
• Visualization: Matplotlib, possibly Seaborn
• Experiment Logging: Weights & Biases (wandb)
• Build Tools: virtualenv, pip