# open-eo/openeo-geopyspark-driver architecture

> Architecture diagram of the GitHub repository open-eo/openeo-geopyspark-driver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-eo/openeo-geopyspark-driver
- Repository: https://github.com/open-eo/openeo-geopyspark-driver

## Overview

##### Main Components

a. Client Layer:
- REST API Interface (Flask-based)
- Direct API Interface (non-REST)
- OpenEO API endpoints

b. Core Processing Layer:
- GeoPySpark/Geotrellis Engine
- Spark Processing Framework
- Batch Job Processing System
- User-Defined Functions (UDF) Handler

c. Storage Layer:
- Accumulo Tile Storage
- Object Storage
- GeoTiff File Storage
- STAC Catalog Integration

d. Infrastructure Components:
- Kubernetes/YARN Cluster
- Zookeeper (for service discovery)
- NginX Reverse Proxy
- Prometheus (monitoring)
- Traefik (routing)

##### Component Relationships

- Client requests → REST/Direct API
- API Layer → Process Graph Executor
- Process Graph Executor → GeoPySpark Engine
- GeoPySpark Engine → Spark Processing
- Spark Processing ↔ Storage Systems
- Infrastructure Services ↔ All Components

##### Specific Implementation Details

Specialized components:
- Batch Job Processing System
- Async Task Handling
- ML Model Integration (CatBoost, Random Forest)
- Vector Cube Processing
- STAC API Integration
- ETL Pipeline

##### Deployment Context

Deployment options:
- Kubernetes Deployment
- YARN Cluster Deployment
- Local Development Setup
- Docker Container Environment

##### Key Integration Points

Integration with:
- Sentinel Hub
- CREO/OSCARS Catalogs
- External Authentication Systems
- Monitoring Systems

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client Layer
    subgraph CL["Client Layer"]
        REST["REST API Interface"]
        DIRECT["Direct API Interface"]
        OPENEO["OpenEO API Endpoints"]
    end

    %% Core Processing Layer
    subgraph CPL["Core Processing Layer"]
        BACKEND["Backend Core"]:::core
        PGE["Process Graph Executor"]
        GPS["GeoPySpark Engine"]
        SPARK["Spark Processing Framework"]
        BATCH["Batch Job System"]:::batch
        UDF["UDF Handler"]:::udf
        ASYNC["Async Task Handler"]:::async
        JOB["Job Tracker"]:::job
        VECTOR["Vector Cube Processing"]:::vector
    end

    %% Storage Layer
    subgraph SL["Storage Layer"]
        ACCUMULO["Accumulo Tile Storage"]
        OBJSTORE["Object Storage"]:::objstorage
        GEOTIFF["GeoTiff Storage"]
        STAC["STAC Integration"]:::stac
    end

    %% Infrastructure Layer
    subgraph IL["Infrastructure Components"]
        K8S["Kubernetes"]:::k8s
        YARN["YARN Cluster"]:::yarn
        ZK["Zookeeper"]
        NGINX["NginX Proxy"]
        PROM["Prometheus"]:::prom
        TRAEFIK["Traefik"]:::traefik
    end

    %% Integration Layer
    subgraph INT["External Integrations"]
        SENT["Sentinel Hub"]:::sentinel
        CREO["CREO/OSCARS Catalogs"]:::catalogs
        ML["ML Models"]:::ml
    end

    %% Relationships
    REST --> BACKEND
    DIRECT --> BACKEND
    OPENEO --> BACKEND
    BACKEND --> PGE
    PGE --> GPS
    GPS --> SPARK
    SPARK --> BATCH
    SPARK --> UDF
    BACKEND --> ASYNC
    ASYNC --> JOB
    SPARK --> VECTOR

    %% Storage Connections
    SPARK --> ACCUMULO
    SPARK --> OBJSTORE
    SPARK --> GEOTIFF
    BACKEND --> STAC

    %% Infrastructure Connections
    K8S -.-> BACKEND
    YARN -.-> SPARK
    ZK -.-> BACKEND
    NGINX -.-> REST
    PROM -.-> BACKEND
    TRAEFIK -.-> REST

    %% Integration Connections
    SENT --> BACKEND
    CREO --> BACKEND
    ML --> SPARK

    %% Styling
    classDef core fill:#f9f,stroke:#333
    classDef batch fill:#fcf,stroke:#333
    classDef udf fill:#cff,stroke:#333
    classDef async fill:#ffc,stroke:#333
    classDef job fill:#cfc,stroke:#333
    classDef vector fill:#ccf,stroke:#333
    classDef objstorage fill:#fcc,stroke:#333
    classDef stac fill:#cff,stroke:#333
    classDef k8s fill:#f9f,stroke:#333
    classDef yarn fill:#fcf,stroke:#333
    classDef prom fill:#ffc,stroke:#333
    classDef traefik fill:#cfc,stroke:#333
    classDef sentinel fill:#ccf,stroke:#333
    classDef catalogs fill:#fcc,stroke:#333
    classDef ml fill:#cff,stroke:#333

    %% Click Events
    click BACKEND "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/backend.py"
    click BATCH "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/deploy/batch_job.py"
    click UDF "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/udf.py"
    click OBJSTORE "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/workspace/object_storage_workspace.py"
    click STAC "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/workspace/stac_api_workspace.py"
    click K8S "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/integrations/kubernetes.py"
    click YARN "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/integrations/yarn.py"
    click PROM "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/integrations/prometheus.py"
    click TRAEFIK "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/integrations/traefik.py"
    click SENT "https://github.com/Open-EO/openeo-geopyspark-driver/tree/master/openeogeotrellis/sentinel_hub"
    click CREO "https://github.com/Open-EO/openeo-geopyspark-driver/tree/master/openeogeotrellis/catalogs"
    click ML "https://github.com/Open-EO/openeo-geopyspark-driver/tree/master/openeogeotrellis/ml"
    click JOB "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/job_tracker_v2.py"
    click ASYNC "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/async_task.py"
    click VECTOR "https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/vectorcube.py"
```
