How shaanisalive/the-machine-learners works
1. What this repo is
- This is an offline, file-driven ML system for monthly urban flood vulnerability forecasting across Bengaluru, Hyderabad, Mumbai, and Pune from 2020-01 to 2024-12.
- The core output is a relative vulnerability score per zone/month, not a flood-depth prediction.
2. Overall architecture
- The repository is organized as a pipeline with clear stages: ingestion → preprocessing → feature construction → model training → inference → evaluation → serving.
- Most stages communicate through artifacts on disk under data/, not via a database or message bus.
3. Data ingestion layer
- src/ingestion/ pulls raw monthly inputs from external geospatial/weather sources and stores them under data/raw/<city>/<source>/<year>/<month>/.
- The raw sources visible in the tree are:
- ERA5 rainfall
- Sentinel-1
- Sentinel-2
- DEM
- GHSL
- OSM roads
- WorldPop
- Each monthly folder contains a manifest.json plus source-specific payloads, so ingestion is both auditable and reproducible.
4. External service boundaries
- The README and configs indicate dependencies on CDS API, Copernicus Data Space (CDSE), and Google Earth Engine.
- These are the only clearly implied external systems; everything after ingestion is local processing of stored artifacts.
5. Preprocessing and flood-stress generation
- src/preprocessing/ turns raw monthly inputs into derived geospatial signals, described in the docs as flood stress signal generation.
- The outputs land in data/processed/, organized by city and month as parquet files.
- This is the first major boundary where heterogeneous source data becomes a standardized tabular/analysis-ready format.
6. Feature assembly
- src/features/dataset_builder.py and data/features/flood_dataset.parquet combine processed signals into the model-ready dataset.
- This layer is where the system aligns monthly environmental, terrain, built-environment, road, and population signals into a single training frame.
7. Modeling and training
- src/models/temporal.py defines the temporal vulnerability model abstraction.
- src/training/pipeline.py orchestrates training and writes metrics/artifacts into data/results/.
- The outputs currently tracked include training_metrics.json and feature_importance.json, which suggests the model is designed to support interpretability as well as prediction.
8. Inference/scoring
- src/inference/pipeline.py uses the trained model to generate vulnerability scores for the configured monthly/city data.
- The primary scored artifact is data/results/vulnerability_scores.parquet, which is the main downstream product consumed by evaluation and serving.
9. Evaluation
- src/evaluation/pipeline.py compares predictions against available reference signals or held-out periods and emits data/results/evaluation.json.
- This is a separate stage from training, which means model quality can be assessed independently of the training run.
10. Serving layer
- src/api/app.py exposes a FastAPI service with endpoints for:
- latest vulnerability
- vulnerability by zone
- vulnerability timeseries
- metadata
- src/frontend/app.py is a Streamlit dashboard for decision-makers, likely consuming the API or the same local result artifacts.
11. Operational entrypoints
- scripts/ provides one command per pipeline/service stage: ingestion, preprocessing, feature build, training, inference, evaluation, API, and frontend.
- This implies the repo is intended to be run as a sequence of batch jobs plus two interactive services.
12. Configuration and environment
- config/ contains city-specific runtime configs, including separate Bengaluru operational settings.
- src/common/settings.py centralizes runtime configuration, while .env supplies credentials and project IDs.
- The system is therefore parameterized by city, time range, and source access credentials rather than hardcoded paths.
13. Data and artifact boundaries
- The main persistent layers are:
- data/raw/ — source-specific monthly raw inputs
- data/processed/ — normalized intermediate outputs
- data/features/ — model dataset
- data/results/ — training, inference, and evaluation outputs
- This makes the architecture diagram naturally layered around disk artifacts, with each stage reading the prior stage’s outputs.
14. Docs and operational knowledge
- docs/ and mkdocs.yml indicate the repo is documented as an operational system, with dedicated pages for ingestion, preprocessing, modeling, inference/scoring, API, UI, and troubleshooting.
- For architecture purposes, the docs reinforce that this is not a single model script but an end-to-end production workflow.
Main entry points: run_training.py Scripts · ingestion/pipeline.py Ingestion · results Results · training/pipeline.py Training
Components 20
Pipeline
Batch stages that transform monthly geospatial inputs into scored vulnerability outputs.
- Ingestion · Pulls monthly raw inputs and writes auditable manifests plus payloads.
src/ingestion/pipeline.py - Preprocess · Turns heterogeneous raw inputs into standardized flood-stress signals.
src/preprocessing/pipeline.py - Dataset build · Aligns monthly signals into the model-ready training frame.
src/features/dataset_builder.py - Temporal model · Defines the temporal scoring abstraction used for vulnerability prediction.
src/models/temporal.py - Training · Orchestrates model fitting and emits training artifacts.
src/training/pipeline.py - Inference · Runs the trained model to produce vulnerability scores.
src/inference/pipeline.py - Evaluation · Compares predictions against reference periods or signals.
src/evaluation/pipeline.py
Disk artifacts
Persistent local layers used to pass data between stages.
- Raw data · Monthly source-specific raw inputs with manifests.
data/raw - Processed data · Normalized monthly intermediate parquet outputs.
data/processed - Feature set · Model-ready merged dataset.
data/features/flood_dataset.parquet - Results · Training, inference, and evaluation outputs.
data/results
Serving
Interactive and API surfaces for consuming the vulnerability scores.
External systems
Upstream geospatial and weather services used during ingestion.
- CDS API · Upstream climate data source used for rainfall ingestion.
- CDSE · Copernicus data access boundary for Sentinel and related assets.
- Earth Engine · Geospatial processing/data access boundary implied by configs and docs.
Config & docs
Runtime configuration, launchers, and operational documentation.
- Settings · Central runtime settings and city/time parameters.
src/common/settings.py - City configs · Per-city operational configuration files.
config/bengaluru_2020_2024.json - Scripts · One-command launchers for pipeline stages and services.
scripts/run_training.py - Docs · Architecture, onboarding, troubleshooting, and usage documentation.
docs/architecture.md
Connections 30
- City configs → Ingestion: configure · City-specific runtime settings drive ingestion behavior.
- Settings → Ingestion: runtime · Shared settings are consumed by ingestion.
- CDS API → Ingestion: rainfall · ERA5 rainfall is fetched from CDS.
- CDSE → Ingestion: imagery · Sentinel and related assets are sourced via Copernicus services.
- Earth Engine → Ingestion: geospatial · Earth Engine is an implied upstream boundary for geospatial access.
- Ingestion → Raw data: writes · Raw monthly payloads and manifests land under data/raw.
- Raw data → Preprocess: inputs · Preprocessing reads the raw monthly artifacts.
- Preprocess → Processed data: writes · Derived monthly flood-stress signals are stored as parquet outputs.
- Processed data → Dataset build: inputs · Feature assembly consumes processed monthly signals.
- Dataset build → Feature set: writes · Builds the model-ready flood dataset.
- Feature set → Training: train data · Training reads the assembled feature set.
- Temporal model → Training: fit · Training instantiates the temporal model abstraction.
- Training → Results: metrics · Training writes metrics and interpretability artifacts.
- Feature set → Inference: score data · Inference scores the model-ready dataset.
- Training → Inference: model · Inference uses the trained model produced by training.
- Inference → Results: scores · Inference emits vulnerability scores parquet.
- Results → Evaluation: predictions · Evaluation consumes scored outputs.
- Feature set → Evaluation: reference · Evaluation can compare against held-out or reference data.
- Evaluation → Results: report · Evaluation writes evaluation.json.
- Results → FastAPI: serve · API reads vulnerability scores and metadata artifacts.
- Results → Streamlit: display · Dashboard consumes result artifacts or the API.
- Scripts → Ingestion: run · Scripts provide stage entrypoints.
- Scripts → Preprocess: run · Scripts provide stage entrypoints.
- Scripts → Training: run · Scripts provide stage entrypoints.
- Scripts → Inference: run · Scripts provide stage entrypoints.
- Scripts → Evaluation: run · Scripts provide stage entrypoints.
- Scripts → FastAPI: run · Scripts provide service entrypoint.
- Scripts → Streamlit: run · Scripts provide service entrypoint.
- Docs → Ingestion: describe · Docs cover the end-to-end workflow.
- Docs → FastAPI: describe · Docs cover API and UI usage.