# shaanisalive/the-machine-learners architecture

> Architecture diagram of the GitHub repository shaanisalive/the-machine-learners, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shaanisalive/the-machine-learners
- Repository: https://github.com/shaanisalive/the-machine-learners

## Overview

1. **What this repo is**
   - This is an offline, file-driven ML system for **monthly urban flood vulnerability forecasting** across **Bengaluru, Hyderabad, Mumbai, and Pune** from **2020-01 to 2024-12**.
   - The core output is a **relative vulnerability score** per zone/month, not a flood-depth prediction.

2. **Overall architecture**
   - The repository is organized as a pipeline with clear stages: **ingestion → preprocessing → feature construction → model training → inference → evaluation → serving**.
   - Most stages communicate through artifacts on disk under `data/`, not via a database or message bus.

3. **Data ingestion layer**
   - `src/ingestion/` pulls raw monthly inputs from external geospatial/weather sources and stores them under `data/raw/<city>/<source>/<year>/<month>/`.
   - The raw sources visible in the tree are:
     - **ERA5 rainfall**
     - **Sentinel-1**
     - **Sentinel-2**
     - **DEM**
     - **GHSL**
     - **OSM roads**
     - **WorldPop**
   - Each monthly folder contains a `manifest.json` plus source-specific payloads, so ingestion is both auditable and reproducible.

4. **External service boundaries**
   - The README and configs indicate dependencies on **CDS API**, **Copernicus Data Space (CDSE)**, and **Google Earth Engine**.
   - These are the only clearly implied external systems; everything after ingestion is local processing of stored artifacts.

5. **Preprocessing and flood-stress generation**
   - `src/preprocessing/` turns raw monthly inputs into derived geospatial signals, described in the docs as **flood stress signal generation**.
   - The outputs land in `data/processed/`, organized by city and month as parquet files.
   - This is the first major boundary where heterogeneous source data becomes a standardized tabular/analysis-ready format.

6. **Feature assembly**
   - `src/features/dataset_builder.py` and `data/features/flood_dataset.parquet` combine processed signals into the model-ready dataset.
   - This layer is where the system aligns monthly environmental, terrain, built-environment, road, and population signals into a single training frame.

7. **Modeling and training**
   - `src/models/temporal.py` defines the temporal vulnerability model abstraction.
   - `src/training/pipeline.py` orchestrates training and writes metrics/artifacts into `data/results/`.
   - The outputs currently tracked include `training_metrics.json` and `feature_importance.json`, which suggests the model is designed to support interpretability as well as prediction.

8. **Inference/scoring**
   - `src/inference/pipeline.py` uses the trained model to generate vulnerability scores for the configured monthly/city data.
   - The primary scored artifact is `data/results/vulnerability_scores.parquet`, which is the main downstream product consumed by evaluation and serving.

9. **Evaluation**
   - `src/evaluation/pipeline.py` compares predictions against available reference signals or held-out periods and emits `data/results/evaluation.json`.
   - This is a separate stage from training, which means model quality can be assessed independently of the training run.

10. **Serving layer**
   - `src/api/app.py` exposes a **FastAPI** service with endpoints for:
     - latest vulnerability
     - vulnerability by zone
     - vulnerability timeseries
     - metadata
   - `src/frontend/app.py` is a **Streamlit** dashboard for decision-makers, likely consuming the API or the same local result artifacts.

11. **Operational entrypoints**
   - `scripts/` provides one command per pipeline/service stage: ingestion, preprocessing, feature build, training, inference, evaluation, API, and frontend.
   - This implies the repo is intended to be run as a sequence of batch jobs plus two interactive services.

12. **Configuration and environment**
   - `config/` contains city-specific runtime configs, including separate Bengaluru operational settings.
   - `src/common/settings.py` centralizes runtime configuration, while `.env` supplies credentials and project IDs.
   - The system is therefore parameterized by city, time range, and source access credentials rather than hardcoded paths.

13. **Data and artifact boundaries**
   - The main persistent layers are:
     - `data/raw/` — source-specific monthly raw inputs
     - `data/processed/` — normalized intermediate outputs
     - `data/features/` — model dataset
     - `data/results/` — training, inference, and evaluation outputs
   - This makes the architecture diagram naturally layered around disk artifacts, with each stage reading the prior stage’s outputs.

14. **Docs and operational knowledge**
   - `docs/` and `mkdocs.yml` indicate the repo is documented as an operational system, with dedicated pages for ingestion, preprocessing, modeling, inference/scoring, API, UI, and troubleshooting.
   - For architecture purposes, the docs reinforce that this is not a single model script but an end-to-end production workflow.

Main entry points: [`scripts/run_training.py`](https://github.com/shaanisalive/the-machine-learners/blob/main/scripts/run_training.py) (Scripts), [`src/ingestion/pipeline.py`](https://github.com/shaanisalive/the-machine-learners/blob/main/src/ingestion/pipeline.py) (Ingestion), [`data/results`](https://github.com/shaanisalive/the-machine-learners/tree/main/data/results) (Results), [`src/training/pipeline.py`](https://github.com/shaanisalive/the-machine-learners/blob/main/src/training/pipeline.py) (Training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_pipeline["Pipeline"]
  node_node_ingestion["Ingestion<br/>raw fetch<br/>[pipeline.py]"]
  node_node_preprocessing["Preprocess<br/>signal derivation<br/>[pipeline.py]"]
  node_node_dataset["Dataset build<br/>feature assembly<br/>[dataset_builder.py]"]
  node_node_model["Temporal model<br/>vulnerability model<br/>[temporal.py]"]
  node_node_training["Training<br/>trainer<br/>[pipeline.py]"]
  node_node_inference["Inference<br/>scoring pipeline<br/>[pipeline.py]"]
  node_node_evaluation["Evaluation<br/>assessment pipeline<br/>[pipeline.py]"]
end

subgraph group_group_storage["Disk artifacts"]
  node_node_raw[("Raw data<br/>artifact layer")]
  node_node_processed[("Processed data<br/>artifact layer")]
  node_node_features[("Feature set<br/>artifact layer")]
  node_node_results[("Results<br/>artifact layer")]
end

subgraph group_group_service["Serving"]
  node_node_api["FastAPI<br/>API service<br/>[app.py]"]
  node_node_frontend["Streamlit<br/>dashboard<br/>[app.py]"]
end

subgraph group_group_external["External systems"]
  node_node_cds(("CDS API<br/>external weather"))
  node_node_cdse(("CDSE<br/>external imagery"))
  node_node_gee(("Earth Engine<br/>external geospatial"))
end

subgraph group_group_ops["Config & docs"]
  node_node_settings["Settings<br/>runtime config<br/>[settings.py]"]
  node_node_ingestion_cfg["City configs"]
  node_node_scripts["Scripts<br/>entrypoints<br/>[run_training.py]"]
  node_node_docs["Docs<br/>[architecture.md]"]
end

node_node_ingestion_cfg -->|"configure"| node_node_ingestion
node_node_settings -->|"runtime"| node_node_ingestion
node_node_cds -.->|"rainfall"| node_node_ingestion
node_node_cdse -.->|"imagery"| node_node_ingestion
node_node_gee -.->|"geospatial"| node_node_ingestion
node_node_ingestion -->|"writes"| node_node_raw
node_node_raw -->|"inputs"| node_node_preprocessing
node_node_preprocessing -->|"writes"| node_node_processed
node_node_processed -->|"inputs"| node_node_dataset
node_node_dataset -->|"writes"| node_node_features
node_node_features -->|"train data"| node_node_training
node_node_model -->|"fit"| node_node_training
node_node_training -->|"metrics"| node_node_results
node_node_features -->|"score data"| node_node_inference
node_node_training -->|"model"| node_node_inference
node_node_inference -->|"scores"| node_node_results
node_node_results -->|"predictions"| node_node_evaluation
node_node_features -->|"reference"| node_node_evaluation
node_node_evaluation -->|"report"| node_node_results
node_node_results -->|"serve"| node_node_api
node_node_results -->|"display"| node_node_frontend
node_node_scripts -->|"run"| node_node_ingestion
node_node_scripts -->|"run"| node_node_preprocessing
node_node_scripts -->|"run"| node_node_training
node_node_scripts -->|"run"| node_node_inference
node_node_scripts -->|"run"| node_node_evaluation
node_node_scripts -->|"run"| node_node_api
node_node_scripts -->|"run"| node_node_frontend
node_node_docs -.->|"describe"| node_node_ingestion
node_node_docs -.->|"describe"| node_node_api

click node_node_settings "https://github.com/shaanisalive/the-machine-learners/blob/main/src/common/settings.py"
click node_node_ingestion_cfg "https://github.com/shaanisalive/the-machine-learners/blob/main/config/bengaluru_2020_2024.json"
click node_node_ingestion "https://github.com/shaanisalive/the-machine-learners/blob/main/src/ingestion/pipeline.py"
click node_node_preprocessing "https://github.com/shaanisalive/the-machine-learners/blob/main/src/preprocessing/pipeline.py"
click node_node_dataset "https://github.com/shaanisalive/the-machine-learners/blob/main/src/features/dataset_builder.py"
click node_node_model "https://github.com/shaanisalive/the-machine-learners/blob/main/src/models/temporal.py"
click node_node_training "https://github.com/shaanisalive/the-machine-learners/blob/main/src/training/pipeline.py"
click node_node_inference "https://github.com/shaanisalive/the-machine-learners/blob/main/src/inference/pipeline.py"
click node_node_evaluation "https://github.com/shaanisalive/the-machine-learners/blob/main/src/evaluation/pipeline.py"
click node_node_api "https://github.com/shaanisalive/the-machine-learners/blob/main/src/api/app.py"
click node_node_frontend "https://github.com/shaanisalive/the-machine-learners/blob/main/src/frontend/app.py"
click node_node_raw "https://github.com/shaanisalive/the-machine-learners/tree/main/data/raw"
click node_node_processed "https://github.com/shaanisalive/the-machine-learners/tree/main/data/processed"
click node_node_features "https://github.com/shaanisalive/the-machine-learners/blob/main/data/features/flood_dataset.parquet"
click node_node_results "https://github.com/shaanisalive/the-machine-learners/tree/main/data/results"
click node_node_scripts "https://github.com/shaanisalive/the-machine-learners/blob/main/scripts/run_training.py"
click node_node_docs "https://github.com/shaanisalive/the-machine-learners/blob/main/docs/architecture.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_ingestion,node_node_preprocessing,node_node_dataset,node_node_model,node_node_training,node_node_inference,node_node_evaluation toneBlue
class node_node_raw,node_node_processed,node_node_features,node_node_results toneAmber
class node_node_api,node_node_frontend toneMint
class node_node_cds,node_node_cdse,node_node_gee toneRose
class node_node_settings,node_node_ingestion_cfg,node_node_scripts,node_node_docs toneIndigo
```

## Components

### Pipeline

Batch stages that transform monthly geospatial inputs into scored vulnerability outputs.

- [Ingestion](https://github.com/shaanisalive/the-machine-learners/blob/main/src/ingestion/pipeline.py): raw fetch · Pulls monthly raw inputs and writes auditable manifests plus payloads. · `src/ingestion/pipeline.py`
- [Preprocess](https://github.com/shaanisalive/the-machine-learners/blob/main/src/preprocessing/pipeline.py): signal derivation · Turns heterogeneous raw inputs into standardized flood-stress signals. · `src/preprocessing/pipeline.py`
- [Dataset build](https://github.com/shaanisalive/the-machine-learners/blob/main/src/features/dataset_builder.py): feature assembly · Aligns monthly signals into the model-ready training frame. · `src/features/dataset_builder.py`
- [Temporal model](https://github.com/shaanisalive/the-machine-learners/blob/main/src/models/temporal.py): vulnerability model · Defines the temporal scoring abstraction used for vulnerability prediction. · `src/models/temporal.py`
- [Training](https://github.com/shaanisalive/the-machine-learners/blob/main/src/training/pipeline.py): trainer · Orchestrates model fitting and emits training artifacts. · `src/training/pipeline.py`
- [Inference](https://github.com/shaanisalive/the-machine-learners/blob/main/src/inference/pipeline.py): scoring pipeline · Runs the trained model to produce vulnerability scores. · `src/inference/pipeline.py`
- [Evaluation](https://github.com/shaanisalive/the-machine-learners/blob/main/src/evaluation/pipeline.py): assessment pipeline · Compares predictions against reference periods or signals. · `src/evaluation/pipeline.py`

### Disk artifacts

Persistent local layers used to pass data between stages.

- [Raw data](https://github.com/shaanisalive/the-machine-learners/tree/main/data/raw): artifact layer · Monthly source-specific raw inputs with manifests. · `data/raw`
- [Processed data](https://github.com/shaanisalive/the-machine-learners/tree/main/data/processed): artifact layer · Normalized monthly intermediate parquet outputs. · `data/processed`
- [Feature set](https://github.com/shaanisalive/the-machine-learners/blob/main/data/features/flood_dataset.parquet): artifact layer · Model-ready merged dataset. · `data/features/flood_dataset.parquet`
- [Results](https://github.com/shaanisalive/the-machine-learners/tree/main/data/results): artifact layer · Training, inference, and evaluation outputs. · `data/results`

### Serving

Interactive and API surfaces for consuming the vulnerability scores.

- [FastAPI](https://github.com/shaanisalive/the-machine-learners/blob/main/src/api/app.py): API service · Serves latest, zonal, timeseries, and metadata endpoints. · `src/api/app.py`
- [Streamlit](https://github.com/shaanisalive/the-machine-learners/blob/main/src/frontend/app.py): dashboard · Decision-facing UI for exploring vulnerability outputs. · `src/frontend/app.py`

### External systems

Upstream geospatial and weather services used during ingestion.

- **CDS API**: external weather · Upstream climate data source used for rainfall ingestion.
- **CDSE**: external imagery · Copernicus data access boundary for Sentinel and related assets.
- **Earth Engine**: external geospatial · Geospatial processing/data access boundary implied by configs and docs.

### Config & docs

Runtime configuration, launchers, and operational documentation.

- [Settings](https://github.com/shaanisalive/the-machine-learners/blob/main/src/common/settings.py): runtime config · Central runtime settings and city/time parameters. · `src/common/settings.py`
- [City configs](https://github.com/shaanisalive/the-machine-learners/blob/main/config/bengaluru_2020_2024.json): city config · Per-city operational configuration files. · `config/bengaluru_2020_2024.json`
- [Scripts](https://github.com/shaanisalive/the-machine-learners/blob/main/scripts/run_training.py): entrypoints · One-command launchers for pipeline stages and services. · `scripts/run_training.py`
- [Docs](https://github.com/shaanisalive/the-machine-learners/blob/main/docs/architecture.md): operational docs · Architecture, onboarding, troubleshooting, and usage documentation. · `docs/architecture.md`

## Connections

- City configs → Ingestion: configure · City-specific runtime settings drive ingestion behavior.
- Settings → Ingestion: runtime · Shared settings are consumed by ingestion.
- CDS API → Ingestion: rainfall · ERA5 rainfall is fetched from CDS.
- CDSE → Ingestion: imagery · Sentinel and related assets are sourced via Copernicus services.
- Earth Engine → Ingestion: geospatial · Earth Engine is an implied upstream boundary for geospatial access.
- Ingestion → Raw data: writes · Raw monthly payloads and manifests land under data/raw.
- Raw data → Preprocess: inputs · Preprocessing reads the raw monthly artifacts.
- Preprocess → Processed data: writes · Derived monthly flood-stress signals are stored as parquet outputs.
- Processed data → Dataset build: inputs · Feature assembly consumes processed monthly signals.
- Dataset build → Feature set: writes · Builds the model-ready flood dataset.
- Feature set → Training: train data · Training reads the assembled feature set.
- Temporal model → Training: fit · Training instantiates the temporal model abstraction.
- Training → Results: metrics · Training writes metrics and interpretability artifacts.
- Feature set → Inference: score data · Inference scores the model-ready dataset.
- Training → Inference: model · Inference uses the trained model produced by training.
- Inference → Results: scores · Inference emits vulnerability scores parquet.
- Results → Evaluation: predictions · Evaluation consumes scored outputs.
- Feature set → Evaluation: reference · Evaluation can compare against held-out or reference data.
- Evaluation → Results: report · Evaluation writes evaluation.json.
- Results → FastAPI: serve · API reads vulnerability scores and metadata artifacts.
- Results → Streamlit: display · Dashboard consumes result artifacts or the API.
- Scripts → Ingestion: run · Scripts provide stage entrypoints.
- Scripts → Preprocess: run · Scripts provide stage entrypoints.
- Scripts → Training: run · Scripts provide stage entrypoints.
- Scripts → Inference: run · Scripts provide stage entrypoints.
- Scripts → Evaluation: run · Scripts provide stage entrypoints.
- Scripts → FastAPI: run · Scripts provide service entrypoint.
- Scripts → Streamlit: run · Scripts provide service entrypoint.
- Docs → Ingestion: describe · Docs cover the end-to-end workflow.
- Docs → FastAPI: describe · Docs cover API and UI usage.