# aitorvv/ml_individual_tree_mortality architecture

> Architecture diagram of the GitHub repository aitorvv/ml_individual_tree_mortality, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aitorvv/ml_individual_tree_mortality
- Repository: https://github.com/aitorvv/ml_individual_tree_mortality

## Overview

The project is an extensive analysis pipeline dedicated to predicting individual tree mortality by comparing Logistic Regression to several machine learning algorithms. The architecture is designed as a sequential data processing and analysis pipeline that ingests raw data, curates and transforms it, runs different analytical and modeling scripts, and finally compiles metrics and produces figures for publication.

##### Project’s Main Purpose and Layers
- Purpose: Comparing logistic regression with several ML techniques (decision trees, random forests, naive Bayes, KNN, SVM, and ANN via HPC scripts) to predict tree mortality based on tree inventory, neighborhood, and climate data.
- High-level view: A data-driven modeling pipeline with clear stages (“data ingestion and curation,” “feature engineering”, “model training & evaluation”, and “results visualization and reporting”).

##### Main Components
a. Data Layer:
- Raw Data Sources: Files in 1_data/0_raw (Excel sheets with tree data, coordinates, VF data).
- Intermediate Data: Checkpoints saved under 1_data/tmp_DEN and processed final outputs under 1_data/1_original_df.
- Climate Data: Imported from external sources (e.g., WorldClim, referenced in the README and handled in the “2_climate_data.r” script).
b. Data Processing / Feature Engineering:
- Data Curation: Script 0_data_curation.r handles initial cleaning and formatting.
- Neighborhood Analysis: Scripts (1.0_neighborhood_main.r, 1.1_neighborhood_functions.r) calculate spatial and neighborhood metrics.
- Climate Integration: Script 2_climate_data.r processes and integrates external climate datasets.
c. Dataset Preparation:
- Splitting and Variable Combinations: Scripts 4.0_split_dataset.r, 4.1_split_variables.r, and 4.2_functions_var_combis.r create multiple case study datasets (by size, thinning, variable set, etc.).
d. Model Training & Analysis:
- Core Modeling Scripts: Group of scripts in section 5 (e.g., 5.0_run_analysis.r through 5.6_SVM_analysis.r) run analyses for each algorithm.
- HPC Scripts: The entire subfolder 2_code/6_HPC shows scripts optimized for high-performance computing setups, supporting parallel training and simulations.
- Metrics Calculation: Metrics and models produced are stored in both case-specific folders (e.g., under 1_data/1_original_df/5_analysis and 6_final_results) and later compiled by scripts like 7_metrics_compilation.r.
e. Visualization & Reporting:
- Graph Generation: Located in 2_code (scripts 8.0_... through 8.6_...) and 9.* scripts produce performance graphs, spatial maps, and paper tables.
- Figures Folder: 3_figures contains final and temporary figures used for analysis and publication.
f. Supporting Components:
- Bibliography and Documentation: 4_bibliography and README.md provide reference information.
- Licensing and Citation: Files like CITATION.cff and LICENSE ensure project compliance and credit.

##### Data Flow and Relationships
- Processed data transitions into the dataset splitting and variable subset creation stage.
- The prepared datasets feed into model training, which is executed via both local R scripts and HPC scripts, producing different case study results.
- The output from the modeling phase (metrics, models, and analysis results) is then compiled for performance evaluation and visualized in graphs and figures.
- Feedback loop where exploratory visualizations (in scripts like 3_feature_visualization.r) might inform adjustments in preprocessing or model configuration.

##### Architectural Patterns and Technologies
- Pipeline Architecture: A sequential “data incoming → processing → modeling → visualization” pattern.
- Modular Design: Different R scripts serve as independent modules that each handle a specific aspect of the workflow.
- Batch and HPC Execution: Note the separation of code intended for standard execution versus high-performance computing (HPC), which should be shown as an optional parallel branch.
- Technologies:
- R as the main programming language.
- Excel and CSV for data exchange.
- HPC (including Python scripts for ANN analysis) for computationally intensive analyses.
- Visualization libraries in R (e.g., ggplot2) and possibly spatial mapping tools.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion
    subgraph "Data Ingestion"
        RD("Raw Data Sources"):::dataLayer
    end

    %% Data Processing
    subgraph "Data Processing"
        DCM("Data Curation Module"):::processModule
        TMP("Intermediate Data (tmp_DEN)"):::dataLayer
        OPD("Processed Data (1_original_df)"):::dataLayer
    end

    %% Feature Engineering
    subgraph "Feature Engineering"
        NAM("Neighborhood Analysis Main"):::processModule
        NAF("Neighborhood Analysis Functions"):::processModule
        CDI("Climate Data Integration"):::processModule
    end

    %% Dataset Preparation
    subgraph "Dataset Preparation"
        SD("Split Dataset"):::processModule
        SV("Split Variables"):::processModule
        FVC("Functions Var Combinations"):::processModule
    end

    %% Model Training & Analysis
    subgraph "Model Training & Analysis"
        subgraph "Local Model Training"
            RA("Run Analysis"):::modelModule
            LRA("LR Analysis"):::modelModule
            DTA("DT Analysis"):::modelModule
            RFA("RF Analysis"):::modelModule
            NBA("NB Analysis"):::modelModule
            KNA("KNN Analysis"):::modelModule
            SVA("SVM Analysis"):::modelModule
        end
        HPC("HPC Model Training"):::modelModule
    end

    %% Metrics & Results
    subgraph "Metrics & Results"
        MC("Metrics Compilation"):::metrics
        M5("Analysis Outputs: 5_analysis"):::metrics
        M6("Analysis Outputs: 6_final_results"):::metrics
    end

    %% Visualization & Reporting
    subgraph "Visualization & Reporting"
        PG("Performance Graphs"):::visualization
        LM("Location Map"):::visualization
        NG("Neighbour Graph"):::visualization
        MG("Mortality Graphs"):::visualization
        DMR("DF Mortality Rates"):::visualization
        PT("Paper Tables"):::visualization
        FD("Figures Directory"):::visualization
    end

    %% Supporting Documentation & Licensing
    subgraph "Supporting"
        DB("Documentation & Bibliography"):::support
        RDOC("README"):::support
        LIC("License"):::support
        CIT("Citation"):::support
    end

    %% Data Flow Connections
    RD -->|"ingests"| DCM
    DCM -->|"produces"| TMP
    DCM -->|"produces"| OPD
    OPD -->|"feeds"| NAM
    OPD -->|"feeds"| NAF
    OPD -->|"feeds"| CDI
    NAM -->|"contributes"| SD
    NAF -->|"contributes"| SD
    CDI -->|"contributes"| SD
    SD -->|"passes"| SV
    SV -->|"passes"| FVC
    FVC -->|"triggers"| RA
    FVC -->|"triggers"| HPC
    RA -->|"pipeline"| LRA
    LRA -->|"pipeline"| DTA
    DTA -->|"pipeline"| RFA
    RFA -->|"pipeline"| NBA
    NBA -->|"pipeline"| KNA
    KNA -->|"pipeline"| SVA
    SVA -->|"outputs"| MC
    HPC -->|"outputs"| MC
    MC -->|"saves to"| M5
    MC -->|"saves to"| M6
    MC -->|"feeds"| PG
    PG -->|"generates"| LM
    LM -->|"generates"| NG
    NG -->|"generates"| MG
    MG -->|"generates"| DMR
    DMR -->|"generates"| PT
    PT -->|"finalizes"| FD

    %% Styling Classes
    classDef dataLayer fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef processModule fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef modelModule fill:#ccccff,stroke:#333,stroke-width:2px;
    classDef metrics fill:#fff2cc,stroke:#333,stroke-width:2px;
    classDef visualization fill:#d9ead3,stroke:#333,stroke-width:2px;
    classDef support fill:#f4cccc,stroke:#333,stroke-width:2px;

    %% Click Events
    click RD "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/1_data/0_raw"
    click DCM "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/0_data_curation.R"
    click TMP "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/1_data/tmp_DEN"
    click OPD "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/1_data/1_original_df"
    click NAM "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/1.0_neighborhood_main.R"
    click NAF "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/1.1_neighborhood_functions.R"
    click CDI "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/2_climate_data.R"
    click SD "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/4.0_split_dataset.r"
    click SV "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/4.1_split_variables.r"
    click FVC "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/4.2_functions_var_combis.r"
    click RA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.0_run_analysis.r"
    click LRA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.1_LR_analysis.r"
    click DTA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.2_DT_analysis.r"
    click RFA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.3_RF_analysis.r"
    click NBA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.4_NB_analysis.r"
    click KNA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.5_KNN_analysis.r"
    click SVA "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/5.6_SVM_analysis.r"
    click HPC "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/2_code/6_HPC"
    click MC "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/7_metrics_compilation.r"
    click M5 "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/1_data/1_original_df/5_analysis"
    click M6 "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/1_data/1_original_df/6_final_results"
    click PG "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/8.0_performance_graphs.r"
    click LM "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/9.0_location_map.R"
    click NG "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/9.1_neighbour_graph.R"
    click MG "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/9.2_mortality_graphs.r"
    click DMR "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/9.3_df_mortality_rates.r"
    click PT "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/2_code/9.4_paper_tables.r"
    click FD "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/3_figures"
    click DB "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/4_bibliography"
    click RDOC "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/README.md"
    click LIC "https://github.com/aitorvv/ml_individual_tree_mortality/tree/main/LICENSE"
    click CIT "https://github.com/aitorvv/ml_individual_tree_mortality/blob/main/CITATION.cff"
```
