How koleshjr/inundata-mapping-floods-in-south-africa works
Main Components
• Data Ingestion & EDA: Represent the initial stage that handles importing composite satellite images, performing Exploratory Data Analysis, and preparing the data. This corresponds to the “first-stage-static-images-eda-data-preparation.ipynb” notebook.
• Image Classification Module: Illustrate the standalone component that uses moisture stress imagery to train an image classifier (using the “eva02_tiny_patch14_224” model) to estimate flood probabilities. This is detailed in “second-stage-image-classifier.ipynb.”
• Modeling Suite: This module should be composite of several subcomponents:
– Various models such as XGBoost, LightGBM, FastAI Tabular, FastAI GatedConv, FastAI 1DConv, FastAI TabTransformer, and TabNet.
– Deep learning models for sequence or time series data like Wavenet-GRU and ResNet1D, with separate training and inference notebooks.
Each of these 9 model approaches can be shown as parallel or a grouped block, indicating they leverage the outputs from the Image Classification Module and additional features (e.g., lagged precipitation, EWMA, etc.).
• Ensembling: Represent the final stage where predictions from all the models are combined using Nelder-Mead optimization and flood probability normalization to achieve the final output. This corresponds to “fourth-stage-models-ensemble.ipynb.”
Relationships and Interactions
• Finally, illustrate the aggregation/ensemble block that takes predictions from all the models and outputs the final, normalized prediction.
Architectural Patterns and Design Principles
• Pipeline/Multi-Stage Architecture: Emphasize the sequential nature of the workflow.
• Modularity: Each notebook (and thus each stage of the pipeline) acts as a module that is invoked in sequence. This modular design makes it easier to update or replace any stage.
• Parallel Modeling: Within the modeling stage, note the simultaneous training of several distinct models, supporting ensemble learning.
• End-to-End Processing: The design supports an end-to-end process from data preparation to ensemble predictions, ensuring traceable and reproducible experimentation.
Technologies and Environment Details
• Explicitly mention that experiments are tailored for execution on Kaggle (or Colab with path modifications), so the environment should be labeled.
• Frameworks employed (e.g., XGBoost, LightGBM, FastAI, TabNet, Wavenet, ResNet1D) as part of the modeling components.
• Notation for the image classification model “eva02_tiny_patch14_224” indicates the usage of a transformer-based architecture for image analysis.
Elements
• Clear labels for each component: Data Preparation, Image Classification, Multiple Modeling Modules, and Ensembling.
• Annotations to specify that the system runs in a notebook-based pipeline on Kaggle (or Colab) and to highlight the main frameworks used in each stage.
• Flow into the image classification module (highlighting the use of “Moisture Stress” as a key feature),
• Branch into multiple parallel model training/inference blocks,
• Converge into an ensembling stage that finalizes the predictions.
This detailed breakdown will help ensure that every architectural element—from dependency flows to underlying model frameworks—is captured accurately, providing a clear and comprehensive overall view of the project’s design for any technical stakeholder evaluating the system.
Happy diagramming!