# s-kganz/forestlst architecture

> Architecture diagram of the GitHub repository s-kganz/forestlst, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/s-kganz/forestlst
- Repository: https://github.com/s-kganz/forestlst

## Overview

##### Main System Components

a) Data Acquisition Layer
- External data sources:
- Aerial Detection Surveys (ADS)
- Earth Engine data
- VODCA data
- Daymet data
- MTBS (Monitoring Trends in Burn Severity)
- NLCD (National Land Cover Database)
- Terrain data

b) Data Processing Layer
- Two parallel pipelines:
- CA_MORT Pipeline (4km resolution)
- WEST_MORT Pipeline (1km resolution)
- Data cleaning and transformation steps

c) Model Development Layer
- Multiple model implementations:
- Neural Networks (PyTorch and TensorFlow)
- AutoML components
- GAM (Generalized Additive Models)
- Preisler CNN implementation

d) Analysis & Evaluation Layer
- Computational Time Analysis
- SHAP Analysis
- Model Assessment
- Training Benchmarks

##### Technology Stack Indicators

- Python ecosystem (main implementation)
- R (for GAM analysis)
- GDAL (for geospatial processing)
- Cloud Services:
- Google Cloud Storage
- Earth Engine
- Earth Access

##### Storage Components

Different storage formats:
- NetCDF files
- TFRecord files
- Zarr arrays
- CSV files
- Geodatabases

##### Specific Implementation Details

Create subsections showing:
- Data resolution differences (4km vs 1km)
- Processing pipeline steps
- Model architecture variations
- Utility functions and their relationships

##### Important Notes

Callouts for:
- Authentication requirements
- Environmental dependencies
- Resource requirements
- Processing sequence dependencies

Key Considerations:
- Modular nature of the utility functions
- Clear separation between data preparation and model training
- Different scales of operation between CA_MORT and WEST_MORT
- Feedback loops for model optimization and evaluation
- Integration points with external services and data sources

This structure will provide a comprehensive view of the system while maintaining clarity about the project's complex data processing and modeling workflows.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Acquisition Layer"
        EE["Earth Engine Interface"]:::data
        EA["Earth Access"]:::data
        DD["Data Downloads"]:::data
        subgraph "Data Sources"
            ADS["ADS Data"]:::source
            EarthEngine["Earth Engine Data"]:::source
            VODCA["VODCA Data"]:::source
            Daymet["Daymet Data"]:::source
            MTBS["MTBS Data"]:::source
            NLCD["NLCD Data"]:::source
            Terrain["Terrain Data"]:::source
        end
    end

    subgraph "Data Processing Layer"
        subgraph "CA_MORT Pipeline"
            CA["CA_MORT Processing (4km)"]:::process
            CA_Clean["Data Cleaning"]:::process
            CA_Transform["Transformations"]:::process
        end
        
        subgraph "WEST_MORT Pipeline"
            WEST["WEST_MORT Processing (1km)"]:::process
            WEST_Clean["Data Cleaning"]:::process
            WEST_Transform["Transformations"]:::process
        end
        
        DPU["Data Processing Utilities"]:::util
        ADS_Proc["ADS Processing"]:::process
        Tensor["Tensor Creation"]:::process
    end

    subgraph "Model Development Layer"
        PT["PyTorch Training"]:::model
        TF["TensorFlow Training"]:::model
        CLSTM["ConvLSTM"]:::model
        AutoML["AutoML"]:::model
        GAM["GAM Models"]:::model
        PCNN["Preisler CNN"]:::model
    end

    subgraph "Analysis & Evaluation Layer"
        CTA["Computational Time Analysis"]:::analysis
        SHAP["SHAP Analysis"]:::analysis
        MA["Model Assessment"]:::analysis
        TB["Training Benchmarks"]:::analysis
        MT["Multi-model Training"]:::analysis
    end

    subgraph "Storage & Constants"
        GCS["Google Cloud Storage"]:::storage
        Config["Constants & Config"]:::storage
        TFR["TFRecord Inspection"]:::storage
        NetCDF["NetCDF Handling"]:::storage
    end

    %% Data Flow Relationships
    EE --> DPU
    EA --> DPU
    DD --> DPU
    ADS --> ADS_Proc
    EarthEngine --> DPU
    VODCA --> DPU
    Daymet --> DPU
    MTBS --> DPU
    NLCD --> DPU
    Terrain --> DPU

    DPU --> CA
    DPU --> WEST
    CA --> CA_Clean --> CA_Transform
    WEST --> WEST_Clean --> WEST_Transform
    
    CA_Transform --> Tensor
    WEST_Transform --> Tensor
    ADS_Proc --> Tensor

    Tensor --> PT
    Tensor --> TF
    Tensor --> CLSTM
    Tensor --> AutoML
    Tensor --> GAM
    Tensor --> PCNN

    PT --> CTA & SHAP & MA & TB & MT
    TF --> CTA & SHAP & MA & TB & MT
    CLSTM --> CTA & SHAP & MA & TB & MT
    AutoML --> CTA & SHAP & MA & TB & MT
    GAM --> CTA & SHAP & MA & TB & MT
    PCNN --> CTA & SHAP & MA & TB & MT

    %% Storage Interactions
    GCS -.-> DPU
    Config -.-> DPU
    Tensor -.-> TFR
    Tensor -.-> NetCDF

    %% Click Events
    click EE "https://github.com/s-kganz/ForestLST/blob/main/util/gee.py"
    click EA "https://github.com/s-kganz/ForestLST/blob/main/notebooks/earthaccess_demo.ipynb"
    click DD "https://github.com/s-kganz/ForestLST/blob/main/scripts/west_mort/download_data.sh"
    click DPU "https://github.com/s-kganz/ForestLST/blob/main/util/data.py"
    click ADS_Proc "https://github.com/s-kganz/ForestLST/blob/main/notebooks/make_ads_images.ipynb"
    click Tensor "https://github.com/s-kganz/ForestLST/blob/main/notebooks/make_tensors.ipynb"
    click PT "https://github.com/s-kganz/ForestLST/blob/main/util/training_torch.py"
    click TF "https://github.com/s-kganz/ForestLST/blob/main/util/training_tf.py"
    click CLSTM "https://github.com/s-kganz/ForestLST/blob/main/util/convlstm.py"
    click AutoML "https://github.com/s-kganz/ForestLST/blob/main/notebooks/AutoML_hyperparameter_tuning.ipynb"
    click GAM "https://github.com/s-kganz/ForestLST/blob/main/notebooks/gam_version.R"
    click PCNN "https://github.com/s-kganz/ForestLST/blob/main/notebooks/preisler_convnet.ipynb"
    click CTA "https://github.com/s-kganz/ForestLST/blob/main/notebooks/Computational_Time_Analysis.ipynb"
    click SHAP "https://github.com/s-kganz/ForestLST/blob/main/notebooks/shap.ipynb"
    click MA "https://github.com/s-kganz/ForestLST/blob/main/notebooks/Model_Training_Assessment.ipynb"
    click TB "https://github.com/s-kganz/ForestLST/blob/main/scripts/west_mort/training_benchmark.py"
    click MT "https://github.com/s-kganz/ForestLST/blob/main/notebooks/multitrain.ipynb"
    click GCS "https://github.com/s-kganz/ForestLST/blob/main/util/gcs.py"
    click Config "https://github.com/s-kganz/ForestLST/blob/main/util/const.py"
    click TFR "https://github.com/s-kganz/ForestLST/blob/main/notebooks/inspect_tfrecords.ipynb"
    click NetCDF "https://github.com/s-kganz/ForestLST/blob/main/notebooks/combine_netcdfs.ipynb"
    click CA "https://github.com/s-kganz/ForestLST/tree/main/scripts/ca_mort/"
    click WEST "https://github.com/s-kganz/ForestLST/tree/main/scripts/west_mort/"

    %% Styling
    classDef data fill:#b3e0ff,stroke:#333,stroke-width:2px
    classDef source fill:#d1c4e9,stroke:#333,stroke-width:2px
    classDef process fill:#c8e6c9,stroke:#333,stroke-width:2px
    classDef util fill:#fff59d,stroke:#333,stroke-width:2px
    classDef model fill:#ffcc80,stroke:#333,stroke-width:2px
    classDef analysis fill:#e1bee7,stroke:#333,stroke-width:2px
    classDef storage fill:#cfd8dc,stroke:#333,stroke-width:2px
```
