# jvargasmu/population_estimation architecture

> Architecture diagram of the GitHub repository jvargasmu/population_estimation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jvargasmu/population_estimation
- Repository: https://github.com/jvargasmu/population_estimation

## Overview

##### Project Type and Purpose
- This is an open-source, research-oriented Python codebase implementing POMELO: a fine-grained population mapping pipeline. It ingests coarse census counts and open geodata (rasters, building footprints), preprocesses them, trains a superpixel-based deep learning model (PyTorch + optional MRF), and outputs high-resolution population maps.

##### File Structure Analysis
- Top-level scripts: data preprocessing (e.g., preprocessing_pop_data.py, preprocessing_zmb_pop_data.py), feature computation (compute_gbuilding_count_map.py, compute_graph.py, compute_percentiles_from_popmap.py), baseline models (building_disagg_baseline.py), model training (superpixel_disagg_model.py, train_model_with_agg_data*.py, train_mrf.py), and visualization (show_feat_importance.py, visualization_*.py).
- Packages:
- cy_utils: Cython extensions for heavy computations (setup.py, .c/.pyx files).
- pix_transform: Python module for pixel-level transforms and evaluation.
- Config: config_pop.py holds all input paths, “no data” values, normalization stats.
- Requirements: requirements.txt specifies dependencies (numpy, torch, gdal, fiona, etc.).

##### Key README Insights
- The pipeline expects georeferenced raster covariates (100 m resolution) and census shapefiles.
- Steps: configure paths in config_pop.py → preprocess census into .pkl → train five cross-validation folds with superpixel_disagg_model.py → ensemble/final inference → generate .tif maps → optional Google Earth Engine export or demo interface.
- Uses WandB for experiment tracking, GDAL/Fiona for I/O, PyTorch for modeling.

##### Main Components
A. Data Sources (Inputs)
- Raster Covariates (WorldPop, night-lights, building counts)
- Census Data & Admin Boundaries (shapefiles, CSV)
B. Configuration
- config_pop.py (paths, norms, no-data values)
C. Preprocessing Module
- preprocessing_pop_data.py → outputs preprocessed census pickle
- compute_gbuilding_count_map.py, compute_graph.py, compute_percentiles_from_popmap.py → feature rasters/arrays
- convert_gbuilding_csv_to_shapefile.py → building footprints
- cy_utils (Cython routines) for compute-heavy steps
D. Pixel Transformation Module
- pix_transform package: transforms raw rasters into model features, evaluation metrics
E. Dataset Preparation
- Collate features + labels into dataset_dir for training
F. Training Module
- superpixel_disagg_model.py (five-fold cross-val & final ensemble)
- train_mrf.py (optional Markov Random Field model)
- train_model_with_agg_data*.py (alternative aggregated-data training)
- Underlying: PyTorch model, GPU acceleration
G. Experiment Tracking & Checkpoints
- WandB integration, checkpoints/ directory
H. Inference & Map Generation
- superpixel_disagg_model.py with “-e5f” flag → high-res .tif outputs
I. Visualization & Evaluation
- show_feat_importance.py, visualization_of_fine_pred_and_gt.py, visualization_of_ocr_pred_and_gt.py

##### Relationships and Data Flow
- Data Sources → Configuration → Preprocessing → Pixel Transformation → Dataset → Training → Inference → Visualization → Final Maps
- cy_utils and utils.py are cross-cutting libraries imported by preprocessing, transformation, and training scripts.
- Checkpoints feed into the final ensemble step.
- WandB and local logs capture metrics at training time.

##### Architectural Patterns and Principles
- Modular design: each script/package focuses on one responsibility.
- Cross-language extension: Cython (cy_utils) for performance-critical loops.
- Config-driven: config_pop.py centralizes paths and normalization metadata.

##### Technologies & Libraries
- Core: Python 3, NumPy, PyTorch, scikit-learn
- Geospatial I/O: GDAL (via gdal==3.2.1), Fiona, rasterio (implicit via GDAL), shapely
- Cython: compiled extensions in cy_utils
- Visualization: Matplotlib, possibly Seaborn
- Experiment Logging: Weights & Biases (wandb)
- Build Tools: virtualenv, pip

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        R1["Raster Covariates"]:::dataStore
        R2["Census Shapefiles & CSV"]:::dataStore
    end

    %% Configuration
    CFG["config_pop.py"]:::config
    click CFG "https://github.com/jvargasmu/population_estimation/blob/main/config_pop.py"

    %% Preprocessing Module
    subgraph "Preprocessing Module"
        direction TB
        P1["preprocessing_pop_data.py"]:::processing
        P2["preprocessing_zmb_pop_data.py"]:::processing
        P3["compute_gbuilding_count_map.py"]:::processing
        P4["compute_graph.py"]:::processing
        P5["compute_percentiles_from_popmap.py"]:::processing
        P6["convert_gbuilding_csv_to_shapefile.py"]:::processing
        CY["cy_utils/"]:::util
    end
    click P1 "https://github.com/jvargasmu/population_estimation/blob/main/preprocessing_pop_data.py"
    click P2 "https://github.com/jvargasmu/population_estimation/blob/main/preprocessing_zmb_pop_data.py"
    click P3 "https://github.com/jvargasmu/population_estimation/blob/main/compute_gbuilding_count_map.py"
    click P4 "https://github.com/jvargasmu/population_estimation/blob/main/compute_graph.py"
    click P5 "https://github.com/jvargasmu/population_estimation/blob/main/compute_percentiles_from_popmap.py"
    click P6 "https://github.com/jvargasmu/population_estimation/blob/main/convert_gbuilding_csv_to_shapefile.py"
    click CY "https://github.com/jvargasmu/population_estimation/tree/main/cy_utils/"

    %% Pixel Transformation
    PT["pix_transform/"]:::processing
    click PT "https://github.com/jvargasmu/population_estimation/tree/main/pix_transform/"

    %% Dataset Preparation & Utilities
    U["utils.py"]:::util
    click U "https://github.com/jvargasmu/population_estimation/blob/main/utils.py"

    %% Baseline Model
    B["building_disagg_baseline.py"]:::processing
    click B "https://github.com/jvargasmu/population_estimation/blob/main/building_disagg_baseline.py"

    %% Training Module
    subgraph "Training & Inference"
        direction TB
        T1["superpixel_disagg_model.py"]:::training
        T2["train_model_with_agg_data.py"]:::training
        T3["train_model_with_agg_data_for_transfer_task.py"]:::training
        T4["train_mrf.py"]:::training
        CK["checkpoints/"]:::dataStore
    end
    click T1 "https://github.com/jvargasmu/population_estimation/blob/main/superpixel_disagg_model.py"
    click T2 "https://github.com/jvargasmu/population_estimation/blob/main/train_model_with_agg_data.py"
    click T3 "https://github.com/jvargasmu/population_estimation/blob/main/train_model_with_agg_data_for_transfer_task.py"
    click T4 "https://github.com/jvargasmu/population_estimation/blob/main/train_mrf.py"

    %% Experiment Tracking
    WB["Weights & Biases"]:::external

    %% Visualization & Evaluation
    subgraph "Visualization & Evaluation"
        direction TB
        V1["show_feat_importance.py"]:::viz
        V2["visualization_of_fine_pred_and_gt.py"]:::viz
        V3["visualization_of_ocr_pred_and_gt.py"]:::viz
    end
    click V1 "https://github.com/jvargasmu/population_estimation/blob/main/show_feat_importance.py"
    click V2 "https://github.com/jvargasmu/population_estimation/blob/main/visualization_of_fine_pred_and_gt.py"
    click V3 "https://github.com/jvargasmu/population_estimation/blob/main/visualization_of_ocr_pred_and_gt.py"

    %% Outputs
    subgraph "Outputs"
        direction TB
        D1["preprocessed census .pkl"]:::dataStore
        D2["dataset_dir"]:::dataStore
        D3["final population .tif"]:::dataStore
    end

    %% Requirements & Environment
    REQ["requirements.txt"]:::util
    GI[".gitignore"]:::util
    RD["README.md"]:::util
    VS[".vscode/"]:::util
    click REQ "https://github.com/jvargasmu/population_estimation/blob/main/requirements.txt"
    click GI "https://github.com/jvargasmu/population_estimation/blob/main/.gitignore"
    click RD "https://github.com/jvargasmu/population_estimation/blob/main/README.md"
    click VS "https://github.com/jvargasmu/population_estimation/tree/main/.vscode/"

    %% Data Flow
    R1 --> CFG
    R2 --> CFG
    CFG --> P1
    CFG --> P2
    P1 --> D1
    P2 --> D1
    P3 --> D1
    P4 --> D1
    P5 --> D1
    P6 --> D1

    D1 --> PT
    PT --> D2

    D2 --> T1
    D2 --> T2
    D2 --> T3
    D2 --> T4

    T1 --> CK
    T1 --> D3
    T2 --> CK
    T3 --> CK
    T4 --> CK

    CK --> T1

    T1 --> V1
    T1 --> V2
    T1 --> V3
    D3 --> V2
    D3 --> V3

    V1 --> D3

    %% Cross-cutting utilities (dashed)
    CY -.-> P1
    CY -.-> P2
    CY -.-> P3
    CY -.-> P4
    CY -.-> P5
    CY -.-> P6
    CY -.-> PT
    CY -.-> T1
    CY -.-> T2
    CY -.-> T3
    CY -.-> T4

    U -.-> P1
    U -.-> PT
    U -.-> T1
    U -.-> T2
    U -.-> T3
    U -.-> T4

    %% Experiment Tracking
    T1 --> WB
    T2 --> WB
    T3 --> WB
    T4 --> WB

    %% Legend
    subgraph Legend
        direction TB
        LS1["\u25CF Data Store"]:::dataStore
        LS2["\u25CF Processing Module"]:::processing
        LS3["\u25CF Training/Inference"]:::training
        LS4["\u25CF Utilities"]:::util
        LS5["\u25CF Visualization"]:::viz
        LS6["\u25CF Config"]:::config
        LS7["\u25CF External Service"]:::external
    end

    classDef dataStore fill:#b7e4c7,stroke:#2d6a4f,color:#1b4332;
    classDef processing fill:#caf0f8,stroke:#0077b6,color:#023e8a;
    classDef training fill:#ffddd2,stroke:#d00000,color:#9d0208;
    classDef util fill:#e0defc,stroke:#5f259f,color:#3a0ca3;
    classDef viz fill:#f1f3f5,stroke:#adb5bd,color:#495057;
    classDef config fill:#fff3bf,stroke:#f08a5d,color:#d1495b;
    classDef external fill:#f8edeb,stroke:#aaa9ad,color:#6c757d;
```
