How miha5092/air-quality-field-reconstruction works
Purpose and execution surface
This is a Python/Linux research pipeline for reconstructing spatiotemporal air-pollution fields from sparse observations. It trains deterministic and generative models on synthetic simulations, then evaluates transfer to real observations. Primary runnable entry points are experiment modules—especially experiments/real_dataset_evaluation.py—plus model-specific training/hyperparameter runners in src/usage/.
Experiment orchestration
Paper scenarios select models, horizons, noise, tuning, and evaluation regimes. Experiment scripts are the top-level control boundary; cluster shell scripts provide batch/HPC launch variants.
• experiments/real_dataset_evaluation.py
• experiments/fine_tuning.py
• experiments/timestep.py
• cluster/diffusion.sh
Data layer: synthetic fields and real observations
Datasets separate generated/simulation-based training data from real-world sensor data. Voronoi utilities likely support spatial partitioning or sensor-placement representation; dataset utilities centralize preprocessing/shared transforms.
• src/datasets/voronoi_datasets.py
• src/datasets/vitae_dataset.py
• src/datasets/real_obs_dataset.py
• src/utils/voronoi.py
Reconstruction model families
The repository compares several interchangeable reconstruction backends: convolutional models, convolutional LSTM models for temporal structure, ViT-AE-style models, and diffusion-based generative reconstruction. Model implementation is separated from execution loops.
• src/models/vcnn.py
• src/models/vclstm.py
• src/models/vitae.py
• src/models/diffusion.py
Training and hyperparameter paths
Usage modules bind datasets, a selected model family, optimization/training logic, and evaluation. Dedicated diffusion training is distinct from deterministic model training, reflecting different objective/sampling behavior. Hyperparameter drivers exist for VCNN and ViTAE.
• src/usage/vcnn_training.py
• src/usage/vitae_training.py
• src/usage/diffusion_training.py
• src/usage/vcnn_hyper.py
Generative diffusion boundary
Diffusion is architecturally split into the model, training pipeline, and noise schedule. This boundary supports iterative denoising/sampling and the repository’s noise-magnitude experiments, unlike direct deterministic output generation.
• src/models/diffusion.py
• src/utils/training_pipeline_diffusion.py
• src/utils/noise_scheduler.py
• experiments/noise_magnitude.py
Evaluation, baselines, and outputs
Evaluation utilities provide common scoring/pipeline behavior across models and datasets; kriging is a separate spatial-statistical baseline path. Notebooks and load_and_eval.py are analysis/visualization consumers rather than core runtime services.
• src/utils/evaluation.py
• src/utils/evaluation_pipeline.py
• src/usage/kriging.py
• load_and_eval.py
Runtime and infrastructure
The system runs under Python 3 on Linux with dependencies pinned in requirements.txt. It has no indicated web service, database, or cloud dependency. Batch infrastructure is shell-script based under cluster/; persistent artifacts appear to be experiment outputs/plots rather than a managed external storage service.
• requirements.txt
• cluster/
• config/train_model.yaml
Main entry points: evaluation_pipeline.py Evaluation pipeline · vcnn_training.py Deterministic training · diffusion_training.py Diffusion training · real_dataset_evaluation.py Experiment scenarios
Components 24
Experiments
- Experiment scenarios · Select model, horizon, noise, tuning, and evaluation regimes.
experiments/real_dataset_evaluation.py - Fine-tuning experiment
experiments/fine_tuning.py - Timestep experiment
experiments/timestep.py - Noise-magnitude experiment
experiments/noise_magnitude.py - HPC batch launch
cluster/diffusion.sh
Data
- Synthetic field data · Training fields generated from simulations.
src/datasets/vitae_dataset.py - Real sensor observations · Transfer-evaluation data from real observations.
src/datasets/real_obs_dataset.py - Voronoi representation · Spatial partitioning or sensor-placement representation.
src/datasets/voronoi_datasets.py - Voronoi utilities
src/utils/voronoi.py
Reconstruction models
- VCNN
src/models/vcnn.py - VCLSTM
src/models/vclstm.py - ViTAE
src/models/vitae.py - Diffusion model · Iteratively denoises fields rather than directly predicting outputs.
src/models/diffusion.py
Training & evaluation
- Deterministic training · Binds direct reconstruction models, data, optimization, and evaluation.
src/usage/vcnn_training.py - ViTAE training
src/usage/vitae_training.py - VCNN hyperparameter search
src/usage/vcnn_hyper.py - Diffusion training · Uses diffusion-specific objectives and sampling behavior.
src/usage/diffusion_training.py - Diffusion pipeline
src/utils/training_pipeline_diffusion.py - Noise scheduler
src/utils/noise_scheduler.py - Evaluation pipeline · Common scoring and evaluation flow across datasets and models.
src/utils/evaluation_pipeline.py - Evaluation metrics
src/utils/evaluation.py - Kriging baseline
src/usage/kriging.py
Other components
- Python/Linux runtime · Pinned research environment dependencies.
requirements.txt - Result analysis · Loads experiment outputs for analysis and visualization.
load_and_eval.py
Connections 26
- HPC batch launch → Diffusion training: launches
- Experiment scenarios → Deterministic training: runs
- Experiment scenarios → Diffusion training: runs
- Fine-tuning experiment → Deterministic training: fine-tunes
- Timestep experiment → Evaluation pipeline: evaluates horizons
- Noise-magnitude experiment → Diffusion training: varies noise
- Voronoi utilities → Voronoi representation: supports
- Synthetic field data → Deterministic training: training data
- Voronoi representation → Deterministic training: spatial input
- Synthetic field data → Diffusion training: training data
- VCNN → Deterministic training: trained by
- VCLSTM → Deterministic training: trained by
- ViTAE → ViTAE training: trained by
- VCNN hyperparameter search → Deterministic training: tunes
- Diffusion model → Diffusion training: trained by
- Diffusion pipeline → Diffusion training: implements loop
- Noise scheduler → Diffusion pipeline: controls noise steps
- Deterministic training → Evaluation pipeline: evaluates
- ViTAE training → Evaluation pipeline: evaluates
- Diffusion training → Evaluation pipeline: evaluates samples
- Real sensor observations → Evaluation pipeline: transfer evaluation
- Evaluation pipeline → Evaluation metrics: computes
- Real sensor observations → Kriging baseline: baseline input
- Kriging baseline → Evaluation pipeline: compares through
- Evaluation pipeline → Result analysis: produces results
- Python/Linux runtime → Experiment scenarios: executes