# pecanproject/pecan architecture

> Architecture diagram of the GitHub repository pecanproject/pecan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pecanproject/pecan
- Repository: https://github.com/pecanproject/pecan

## Overview

##### Main components
a. Front-end Layer
- Static website / PHP interface (“web/” folder) serving UI pages (PHP/Leaflet)
- Shiny applications (“shiny/” folder) for specialized dashboards (Data-Ingest, Elicitation, ForecastingDashboard, etc.)
- Bookdown/documentation site (“book_source/”)
b. API Layer
- R-based REST API (under “apps/api/” – a Dockerfile plus R scripts using plumber)
c. Workflow Engine & Executors
- Controller scripts (R modules in subfolders under “base/” and “modules/”) that implement data ingest, conversion, assimilation, benchmarking, uncertainty, remote execution
- Message broker / queue service (RabbitMQ code in “base/remote/R” and in “docker/executor” / “docker/monitor”)
- Executor service (Python-based “docker/executor/executor.py”)
d. Model Wrappers & Containers
- A dozen+ ecosystem models, each in “models/<model_name>/” with its own Dockerfile or source wrapper
- Shared R helper packages in “base/” (db, logger, utils, settings, qaqc, visualization, workflow) and “modules/” (assimilation, data.atmosphere, data.land, data.remote, emulator, meta.analysis, rtm, uncertainty, etc.)
e. Data Store
- PostgreSQL (BETYDB) for metadata, traits, workflow records
- File store for intermediate netCDF, XML configurations, logs
- Docker volumes for persisting DB, file artifacts
f. Orchestration & Deployment
- Docker Compose configurations (“docker-compose.yml” variants)
- CI workflows (.github/workflows)
- Makefiles, scripts for local builds, CI, deployments

##### Relationships & flow
- User → Web UI (PHP pages / Shiny apps)
→ sends HTTP (or WebSocket) requests → API
- API
→ queries DB (Postgres) for metadata, settings
→ stores workflow requests in DB
→ publishes tasks to RabbitMQ
- Executor service
→ subscribes to RabbitMQ
→ pulls job descriptor → spins up model container (via Docker or submits remote HPC job)
→ returns status & output files → writes metadata/results to DB and file store
- Front-end dashboards
→ poll API or directly read DB/file store for job status, visualizations

##### Architectural patterns
- Microservices containerized with Docker
- Message-driven workflow (RabbitMQ) between API and executor
- Modular R packages exposing functionality via runModule.* functions
- Separation of concerns: web vs API vs executor vs model wrappers vs data stores

##### Technologies
- R / plumber (API), PHP (web UI), Shiny (dashboards)
- PostgreSQL (BETYDB)
- RabbitMQ (messaging)
- Docker & Docker Compose (containerization)
- Continuous integration (GitHub Actions)

##### Essential details
- Docker Compose “stack” showing how services link (web, api, db, rabbitmq, executor, monitor)
- R-package modules feeding into the API and executor layers
- Distinction between local Docker-based execution and remote HPC via base/remote

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User"]:::ui

    %% Front-end Layer
    subgraph "Front-end Layer"
        PHPUI["PHP Web UI"]:::ui
        Shiny["Shiny Apps"]:::ui
        Docs["Documentation Site"]:::ui
    end

    %% API & Workflow Services
    subgraph "API & Workflow Services"
        API["R-based REST API"]:::service
        CoreModules["Core R Modules (base/)"]:::service
        SpecModules["Specialized R Modules (modules/)"]:::service
        ExecutorSvc["Executor Service (Python)"]:::service
        Monitor["Monitor Service"]:::service
    end

    %% Data Stores & Messaging
    subgraph "Data Stores & Messaging"
        DB["PostgreSQL (BETYDB)"]:::data
        FileStore["File Store"]:::data
        Rabbit["RabbitMQ"]:::data
    end

    %% Model Container Pool
    subgraph "Model Container Pool"
        ModelImages["Ecosystem Model Images"]:::model
    end

    %% Orchestration & Deployment
    subgraph "Orchestration & Deployment"
        Compose["Docker Compose"]:::orches
        CI["CI (GitHub Actions)"]:::orches
        Scripts["Utility Scripts"]:::orches
    end

    %% Connections
    User --> PHPUI
    User --> Shiny
    User --> Docs

    PHPUI -->|"HTTP"| API
    Shiny -->|"HTTP"| API
    Docs -->|"HTTP"| API

    API -->|"SQL"| DB
    API -->|"store request"| DB
    API -->|"publish task (AMQP)"| Rabbit

    CoreModules --> API
    SpecModules --> API

    ExecutorSvc -->|"subscribe (AMQP)"| Rabbit
    ExecutorSvc -->|"spin up container"| ModelImages
    ExecutorSvc -->|"write results"| DB
    ExecutorSvc -->|"write artifacts"| FileStore

    Monitor -->|"monitor queue"| Rabbit

    Shiny -->|"poll status"| API
    Shiny -->|"read data"| FileStore

    %% Orchestration links
    Compose --> PHPUI
    Compose --> Shiny
    Compose --> API
    Compose --> Rabbit
    Compose --> DB
    Compose --> ExecutorSvc
    Compose --> Monitor

    CI --> Compose
    Scripts --> Compose

    %% Click Events
    click PHPUI "https://github.com/pecanproject/pecan/tree/develop/web/"
    click Shiny "https://github.com/pecanproject/pecan/tree/develop/shiny/"
    click Docs "https://github.com/pecanproject/pecan/tree/develop/book_source/"
    click API "https://github.com/pecanproject/pecan/tree/develop/apps/api/"
    click CoreModules "https://github.com/pecanproject/pecan/tree/develop/base/workflow/"
    click SpecModules "https://github.com/pecanproject/pecan/tree/develop/modules/"
    click Rabbit "https://github.com/pecanproject/pecan/tree/develop/base/remote/"
    click ExecutorSvc "https://github.com/pecanproject/pecan/tree/develop/docker/executor/"
    click Monitor "https://github.com/pecanproject/pecan/tree/develop/docker/monitor/"
    click ModelImages "https://github.com/pecanproject/pecan/tree/develop/models/"
    click Compose "https://github.com/pecanproject/pecan/blob/develop/docker-compose.yml"
    click CI "https://github.com/pecanproject/pecan/tree/develop/.github/workflows/"
    click Scripts "https://github.com/pecanproject/pecan/tree/develop/scripts/"

    %% Styles
    classDef ui fill:#D0E6FF,stroke:#0366D6,color:#0366D6
    classDef service fill:#DFF7DF,stroke:#22863A,color:#22863A
    classDef data fill:#FFF5B1,stroke:#DBAB09,color:#735C0F
    classDef model fill:#FFEFEF,stroke:#D73A49,color:#D73A49
    classDef orches fill:#F0F0F0,stroke:#6A737D,color:#6A737D
```
