# ioos/cloud-sandbox architecture

> Architecture diagram of the GitHub repository ioos/cloud-sandbox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ioos/cloud-sandbox
- Repository: https://github.com/ioos/cloud-sandbox

## Overview

##### Main Components

a. Cloud Infrastructure Layer:
- AWS Cloud boundary (outer container)
- VPC (Virtual Private Cloud) container
- Head Node (Control Plane)
- Worker Nodes (Data Plane)
- Storage Services (S3, EFS)
- AMI Repository

b. Software Stack Layer:
- CloudFlow Orchestration System
- Prefect Workflow Engine
- Model Execution Environment
- Various Coastal Models (ROMS, FVCOM, ADCIRC, etc.)

c. External Interfaces:
- User SSH Access
- Data Input Sources
- Results Storage/Output

##### Component Relationships

- User → Head Node (SSH connection)
- Head Node → Worker Nodes (CloudFlow task distribution)
- Worker Nodes → Storage Services (bi-directional for data access)
- Head Node → CloudFlow Orchestrator (control flow)
- CloudFlow → Prefect (workflow execution)
- Worker Nodes → Model Execution (deployment)

##### Important Architectural Elements

Key architectural patterns:
- Scheduler-Worker pattern (CloudFlow)
- Infrastructure as Code (Terraform)
- Container-based deployment
- Distributed storage system

##### Specific Implementation Details

Crucial components from the codebase:
- cloudflow/cluster/ (cluster management)
- cloudflow/job/ (job execution)
- cloudflow/services/ (storage services)
- cloudflow/workflows/ (workflow definitions)

**┌─────────────────────── AWS Cloud ───────────────────────┐**
- ┌─────────────────────── VPC ─────────────────────────┐ │

- ┌─────────┐    ┌─────────┐    ┌─────────┐        │ │
- Head Node│───▶│CloudFlow│───▶│ Prefect │        │ │
- ┘    └─────────┘    └─────────┘        │ │

- ▼                                           │ │
- ┌─────────┐    ┌─────────┐    ┌─────────┐      │ │
- Worker  │◄──▶│ Models  │◄──▶│Storage  │      │ │
- Nodes   │    │Execution│    │Services │      │ │
- ┘    └─────────┘    └─────────┘      │ │
- ┘ │
- ┘

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph AWS["AWS Cloud Environment"]
        subgraph VPC["Virtual Private Cloud"]
            HeadNode["Head Node\nControl Plane"]:::control
            WorkerNodes["Worker Nodes\nData Plane"]:::worker
            
            subgraph Orchestration["Orchestration Layer"]
                CloudFlow["CloudFlow\nOrchestrator"]:::orchestrator
                Prefect["Prefect\nWorkflow Engine"]:::orchestrator
                Tasks["Task Management"]:::orchestrator
            end
            
            subgraph Storage["Storage Services"]
                S3["S3 Storage"]:::storage
                FSx["FSx Storage"]:::storage
                NFS["NFS Storage"]:::storage
            end
            
            subgraph Models["Model Execution Layer"]
                ROMS["ROMS Model"]:::model
                FVCOM["FVCOM Model"]:::model
                ADCIRC["ADCIRC Model"]:::model
            end
            
            subgraph Visualization["Visualization Layer"]
                ROMSPlot["ROMS Plotting"]:::viz
                FVCOMPlot["FVCOM Plotting"]:::viz
            end
        end
        
        AMI["AMI Repository"]:::infra
        TerraformInfra["Infrastructure as Code"]:::infra
    end
    
    User["External User"]:::external
    DataSources["Data Input Sources"]:::external
    Results["Results Storage"]:::external

    %% Relationships
    User --> HeadNode
    DataSources --> Storage
    Storage --> Results
    
    HeadNode --> CloudFlow
    CloudFlow --> Prefect
    Prefect --> Tasks
    
    HeadNode --> WorkerNodes
    WorkerNodes <--> Storage
    
    WorkerNodes --> Models
    Models --> Storage
    
    Models --> Visualization
    Visualization --> Storage
    
    TerraformInfra --> VPC
    AMI --> WorkerNodes

    %% Click Events
    click CloudFlow "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/cluster/Cluster.py"
    click HeadNode "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/cluster/AWSCluster.py"
    click S3 "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/services/S3Storage.py"
    click FSx "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/services/FSxScratchDisk.py"
    click NFS "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/services/NFSScratchDisk.py"
    click Tasks "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/workflows/tasks.py"
    click ROMS "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/job/ROMSForecast.py"
    click FVCOM "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/job/FVCOMForecast.py"
    click ADCIRC "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/job/ADCIRCForecast.py"
    click ROMSPlot "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/plotting/plot_roms.py"
    click FVCOMPlot "https://github.com/ioos/Cloud-Sandbox/blob/main/cloudflow/plotting/plot_fvcom.py"
    click TerraformInfra "https://github.com/ioos/Cloud-Sandbox/blob/main/terraform/main.tf"

    %% Styles
    classDef external fill:#gray,stroke:#333,stroke-width:2px
    classDef control fill:#3498db,stroke:#2980b9,stroke-width:2px
    classDef worker fill:#2ecc71,stroke:#27ae60,stroke-width:2px
    classDef orchestrator fill:#e74c3c,stroke:#c0392b,stroke-width:2px
    classDef storage fill:#9b59b6,stroke:#8e44ad,stroke-width:2px
    classDef model fill:#f1c40f,stroke:#f39c12,stroke-width:2px
    classDef viz fill:#1abc9c,stroke:#16a085,stroke-width:2px
    classDef infra fill:#95a5a6,stroke:#7f8c8d,stroke-width:2px
```
