How ioos/cloud-sandbox works
Main Components
a. Cloud Infrastructure Layer:
• AWS Cloud boundary (outer container)
• VPC (Virtual Private Cloud) container
• Head Node (Control Plane)
• Worker Nodes (Data Plane)
• Storage Services (S3, EFS)
• AMI Repository
b. Software Stack Layer:
• CloudFlow Orchestration System
• Prefect Workflow Engine
• Model Execution Environment
• Various Coastal Models (ROMS, FVCOM, ADCIRC, etc.)
c. External Interfaces:
• User SSH Access
• Data Input Sources
• Results Storage/Output
Component Relationships
• User → Head Node (SSH connection)
• Head Node → Worker Nodes (CloudFlow task distribution)
• Worker Nodes → Storage Services (bi-directional for data access)
• Head Node → CloudFlow Orchestrator (control flow)
• CloudFlow → Prefect (workflow execution)
• Worker Nodes → Model Execution (deployment)
Important Architectural Elements
Key architectural patterns:
• Scheduler-Worker pattern (CloudFlow)
• Infrastructure as Code (Terraform)
• Container-based deployment
• Distributed storage system
Specific Implementation Details
Crucial components from the codebase:
• cloudflow/cluster/ (cluster management)
• cloudflow/job/ (job execution)
• cloudflow/services/ (storage services)
• cloudflow/workflows/ (workflow definitions)
┌─────────────────────── AWS Cloud ───────────────────────┐
• ┌─────────────────────── VPC ─────────────────────────┐ │
• ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
• Head Node│───▶│CloudFlow│───▶│ Prefect │ │ │
• ┘ └─────────┘ └─────────┘ │ │
• ▼ │ │
• ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
• Worker │◄──▶│ Models │◄──▶│Storage │ │ │
• Nodes │ │Execution│ │Services │ │ │
• ┘ └─────────┘ └─────────┘ │ │
• ┘ │
• ┘