# harrisb002/hyperspectral-landcover-classification architecture

> Architecture diagram of the GitHub repository harrisb002/hyperspectral-landcover-classification, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harrisb002/hyperspectral-landcover-classification
- Repository: https://github.com/harrisb002/hyperspectral-landcover-classification

## Overview

##### Project Type & Purpose
- This project is a machine learning pipeline focused on hyperspectral image analysis and land cover classification. It combines heavy data processing (including cleaning, PCA‐based outlier detection, and a 1D CNN for image pixel classification) with a web-based visualization component.
- Additionally, it is part of a larger initiative (BioSCape) and includes a React/Node.js application that visualizes predictions and geospatial data using Google Maps and deck.gl.

##### Analyzing the File Structure
- Python Modules & Scripts:
– Directories like “model”, “pipelines”, “steps”, “materializer” and scripts such as “run_pipeline.py” and “run_deployment.py” indicate separate stages:
▸ Data ingest and preprocessing (steps/ingest_data.py, steps/clean_data.py)
▸ Model development and evaluation (model/model_dev.py, model/evaluation.py)
▸ Utility functions and orchestration (pipelines/utils.py)
▸ Deployment orchestration (run_deployment.py) and checkpoint storage (saved_weights)
- Notebooks:
– Files such as “CombinedCNN.ipynb” and “createDF.ipynb” serve as exploratory work or prototypes for data analysis and model development.
- Web Application:
– The “hsi-visualization” directory contains a Node.js server (server.js) and a React-based front end (inside src/), including components for maps, plots, controls, and hooks to load data.
- Assets:

A. Data and Processing Backend
- Raw Data/Input Sources:
- Data sources such as raw hyperspectral images and Google Earth derived image-level samples.
- Data Preprocessing and Outlier Removal:
- “steps” modules (ingest_data, clean_data) and “model/data_cleaning.py” that perform cleaning, PCA with Mahalanobis distance filtering, and outlier removal.
- Machine Learning Model Training and Evaluation:
- “model” directory components (model_dev, evaluation) and the 1D CNN architecture. Include the process flow: from preprocessed input → CNN layers (with Conv1D, pooling, dense layers) → model predictions.
- Saving of model checkpoints (saved_weights) and the role of the “pipelines” and “materializer” modules.
- Orchestration and Deployment:
- How “run_pipeline.py” coordinates the processing and training steps.

B. Web-Based Visualization Front End
- Node.js Server:
- “hsi-visualization” folder’s server.js that acts as the back-end API server.
- Front-End Application:
- React components (App.js, map components, plot pages) which leverage Google Maps API and deck.gl for geospatial visualization.
- UI components: Sidebar (category filtering), TopBar, MapContainer, and interactive visualizations (bar and scatter plots).
- External Integrations:
- External services like the Google Maps API, deck.gl, and the MongoDB backend (mentioned in the README for pre-aggregated frequency sums).

##### Relationships and Interactions Between Components
- Data flow from raw data sources to the preprocessing modules.
- Processed data being fed to the CNN model for training and evaluation.
- The saved model artifacts (weights) then being accessed by the deployment script.
- The deployed endpoint being consumed by the Node.js server, which in turn supports the React front end.
- Interaction between the data back end and external services (e.g., MongoDB for aggregated geospatial data).

##### Architectural Patterns & Important Principles
- Separation of Concerns:
- Clearly distinguish between the data ingestion/processing logic and the visualization layer.
- Layered Architecture:
- Presentation (React UI), API backend (Node.js server), business and ML logic (Python processing), and data storage (model checkpoints, MongoDB) as separate layers.
- Pipeline Orchestration:
- Sequential steps: ingestion → cleaning → outlier removal → model training → evaluation → deployment.
- Communication and Integration:
- Key libraries/frameworks (e.g., TensorFlow/Keras for the CNN, React and Node.js for the web app, and possibly deck.gl and Google Maps API for geospatial visuals).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data and Processing Backend
    subgraph "Data and Processing Backend"
        RD["Raw Hyperspectral Data"]:::external
        DI["Data Ingestion"]:::python
        CDS["Clean Data Step"]:::python
        DCM["Data Cleaning Module"]:::python
        MD["Model Development"]:::python
        MT["Model Training"]:::python
        ME["Model Evaluation"]:::python
        ES["Evaluation Step"]:::python
        UC["Utility Functions"]:::python
        PO["Pipeline Orchestration"]:::python
        DO["Deployment Orchestration"]:::python
        MC["Model Checkpoint Storage"]:::python
    end

    %% Web-Based Visualization Front End
    subgraph "Web-Based Visualization Front End"
        AS["API Server"]:::web
        FE["Front-End Application"]:::web
        EX["External Services (GoogleMaps,deck.gl,MongoDB)"]:::external
    end

    %% Data Flow in Backend
    RD -->|"raw_hyperspectral_data"| DI
    DI -->|"ingested_data"| CDS
    CDS -->|"cleaned_data"| DCM
    DCM -->|"preprocessed_features"| MD
    MD -->|"development_flow"| MT
    MT -->|"trained_model"| ME
    ME -->|"evaluation_metrics"| ES
    MT -->|"save_checkpoint"| MC
    ES -->|"evaluation_feedback"| PO
    UC -->|"orchestration_support"| PO
    PO -->|"trigger_deployment"| DO
    MC -->|"model_artifact"| DO

    %% Connection from Backend to Front End
    DO -->|"deployed_endpoint"| AS
    AS -->|"api_calls"| FE
    AS -->|"external_data_requests"| EX
    FE -->|"visualization_requests"| EX

    %% Click Events
    click DI "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/steps/ingest_data.py"
    click CDS "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/steps/clean_data.py"
    click DCM "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/model/data_cleaning.py"
    click MD "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/model/model_dev.py"
    click MT "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/steps/model_train.py"
    click ME "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/model/evaluation.py"
    click ES "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/steps/evaluation.py"
    click UC "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/pipelines/utils.py"
    click PO "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/run_pipeline.py"
    click DO "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/run_deployment.py"
    click MC "https://github.com/harrisb002/hyperspectral-landcover-classification/tree/main/saved_weights/"
    click AS "https://github.com/harrisb002/hyperspectral-landcover-classification/blob/main/hsi-visualization/server.js"
    click FE "https://github.com/harrisb002/hyperspectral-landcover-classification/tree/main/hsi-visualization/src/"

    %% Styles
    classDef python fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef web fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef external fill:#FFD700,stroke:#000,stroke-width:1px;
```
