How aitorvv/ml_individual_tree_mortality works
The project is an extensive analysis pipeline dedicated to predicting individual tree mortality by comparing Logistic Regression to several machine learning algorithms. The architecture is designed as a sequential data processing and analysis pipeline that ingests raw data, curates and transforms it, runs different analytical and modeling scripts, and finally compiles metrics and produces figures for publication.
Project’s Main Purpose and Layers
• Purpose: Comparing logistic regression with several ML techniques (decision trees, random forests, naive Bayes, KNN, SVM, and ANN via HPC scripts) to predict tree mortality based on tree inventory, neighborhood, and climate data.
• High-level view: A data-driven modeling pipeline with clear stages (“data ingestion and curation,” “feature engineering”, “model training & evaluation”, and “results visualization and reporting”).
Main Components
a. Data Layer:
• Raw Data Sources: Files in 1_data/0_raw (Excel sheets with tree data, coordinates, VF data).
• Intermediate Data: Checkpoints saved under 1_data/tmp_DEN and processed final outputs under 1_data/1_original_df.
• Climate Data: Imported from external sources (e.g., WorldClim, referenced in the README and handled in the “2_climate_data.r” script).
b. Data Processing / Feature Engineering:
• Data Curation: Script 0_data_curation.r handles initial cleaning and formatting.
• Neighborhood Analysis: Scripts (1.0_neighborhood_main.r, 1.1_neighborhood_functions.r) calculate spatial and neighborhood metrics.
• Climate Integration: Script 2_climate_data.r processes and integrates external climate datasets.
c. Dataset Preparation:
• Splitting and Variable Combinations: Scripts 4.0_split_dataset.r, 4.1_split_variables.r, and 4.2_functions_var_combis.r create multiple case study datasets (by size, thinning, variable set, etc.).
d. Model Training & Analysis:
• Core Modeling Scripts: Group of scripts in section 5 (e.g., 5.0_run_analysis.r through 5.6_SVM_analysis.r) run analyses for each algorithm.
• HPC Scripts: The entire subfolder 2_code/6_HPC shows scripts optimized for high-performance computing setups, supporting parallel training and simulations.
• Metrics Calculation: Metrics and models produced are stored in both case-specific folders (e.g., under 1_data/1_original_df/5_analysis and 6_final_results) and later compiled by scripts like 7_metrics_compilation.r.
e. Visualization & Reporting:
• Graph Generation: Located in 2_code (scripts 8.0_... through 8.6_...) and 9.* scripts produce performance graphs, spatial maps, and paper tables.
• Figures Folder: 3_figures contains final and temporary figures used for analysis and publication.
f. Supporting Components:
• Bibliography and Documentation: 4_bibliography and README.md provide reference information.
• Licensing and Citation: Files like CITATION.cff and LICENSE ensure project compliance and credit.
Data Flow and Relationships
• Processed data transitions into the dataset splitting and variable subset creation stage.
• The prepared datasets feed into model training, which is executed via both local R scripts and HPC scripts, producing different case study results.
• The output from the modeling phase (metrics, models, and analysis results) is then compiled for performance evaluation and visualized in graphs and figures.
• Feedback loop where exploratory visualizations (in scripts like 3_feature_visualization.r) might inform adjustments in preprocessing or model configuration.
Architectural Patterns and Technologies
• Pipeline Architecture: A sequential “data incoming → processing → modeling → visualization” pattern.
• Modular Design: Different R scripts serve as independent modules that each handle a specific aspect of the workflow.
• Batch and HPC Execution: Note the separation of code intended for standard execution versus high-performance computing (HPC), which should be shown as an optional parallel branch.
• Technologies:
• R as the main programming language.
• Excel and CSV for data exchange.
• HPC (including Python scripts for ANN analysis) for computationally intensive analyses.
• Visualization libraries in R (e.g., ggplot2) and possibly spatial mapping tools.