How pecanproject/pecan works
Main components
a. Front-end Layer
• Static website / PHP interface (“web/” folder) serving UI pages (PHP/Leaflet)
• Shiny applications (“shiny/” folder) for specialized dashboards (Data-Ingest, Elicitation, ForecastingDashboard, etc.)
• Bookdown/documentation site (“book_source/”)
b. API Layer
• R-based REST API (under “apps/api/” – a Dockerfile plus R scripts using plumber)
c. Workflow Engine & Executors
• Controller scripts (R modules in subfolders under “base/” and “modules/”) that implement data ingest, conversion, assimilation, benchmarking, uncertainty, remote execution
• Message broker / queue service (RabbitMQ code in “base/remote/R” and in “docker/executor” / “docker/monitor”)
• Executor service (Python-based “docker/executor/executor.py”)
d. Model Wrappers & Containers
• A dozen+ ecosystem models, each in “models/<model_name>/” with its own Dockerfile or source wrapper
• Shared R helper packages in “base/” (db, logger, utils, settings, qaqc, visualization, workflow) and “modules/” (assimilation, data.atmosphere, data.land, data.remote, emulator, meta.analysis, rtm, uncertainty, etc.)
e. Data Store
• PostgreSQL (BETYDB) for metadata, traits, workflow records
• File store for intermediate netCDF, XML configurations, logs
• Docker volumes for persisting DB, file artifacts
f. Orchestration & Deployment
• Docker Compose configurations (“docker-compose.yml” variants)
• CI workflows (.github/workflows)
• Makefiles, scripts for local builds, CI, deployments
Relationships & flow
• User → Web UI (PHP pages / Shiny apps)
→ sends HTTP (or WebSocket) requests → API
• API
→ queries DB (Postgres) for metadata, settings
→ stores workflow requests in DB
→ publishes tasks to RabbitMQ
• Executor service
→ subscribes to RabbitMQ
→ pulls job descriptor → spins up model container (via Docker or submits remote HPC job)
→ returns status & output files → writes metadata/results to DB and file store
• Front-end dashboards
→ poll API or directly read DB/file store for job status, visualizations
Architectural patterns
• Microservices containerized with Docker
• Message-driven workflow (RabbitMQ) between API and executor
• Modular R packages exposing functionality via runModule.* functions
• Separation of concerns: web vs API vs executor vs model wrappers vs data stores
Technologies
• R / plumber (API), PHP (web UI), Shiny (dashboards)
• PostgreSQL (BETYDB)
• RabbitMQ (messaging)
• Docker & Docker Compose (containerization)
• Continuous integration (GitHub Actions)
Essential details
• Docker Compose “stack” showing how services link (web, api, db, rabbitmq, executor, monitor)
• R-package modules feeding into the API and executor layers
• Distinction between local Docker-based execution and remote HPC via base/remote