How novonordisk-opensource/whirl works
Project type and purpose
• This is an R package (“whirl”) whose purpose is to batch-execute R (and Python) scripts under isolated sessions, capture their status/environment/logs, and render per-script and summary HTML reports. It’s a command-line / library tool, not a web app or microservice.
File structure
• Top level
• DESCRIPTION, NAMESPACE, R/, inst/, man/, tests/, vignettes/, .github/
• R/: all core code broken into many R modules (run.R, internal_run.R, custom_logging.R, utils.R, queue management, session setup, mdformats/quarto wrappers, etc.)
• inst/: shipped assets – QMD templates, example YAML configs, Python helper modules, RStudio addins.
• man/: .Rd docs per public function.
• tests/: testthat tests and snapshot files drive CI.
• vignettes/: user guides (quarto/Rmd).
• .github/: CI workflows for build & test.
README for additional insights
• Main entrypoint: run(input, n_workers, …)
• Supports either script paths or a YAML “steps” config.
• Spins up parallel workers, executes scripts in fresh R sessions, logs code, status, env info, and turns logs into HTML via mdformats/quarto.
• Produces per-script “_log.html” and an aggregated “summary.html.”
• Offers customizable hooks via whirl-options and renv integration.
1. User/API Layer
– R console CLI or RStudio addin calling whirl::run()
2. Config Parser
– Reads script paths or YAML steps file (inst/use_whirl/_whirl.yml patterns)
3. Scheduler / Worker Pool
– Manages n_workers, enqueues jobs, dequeues when workers available
4. Execution Engine
– For each script:
• Launches isolated R session (Rscript) via whirl_r_session
• Passes script to internal_run
5. Logging Module
– custom_logging captures stdout, warnings, errors, session info
– mdformats & quarto wrappers render HTML (inst/documents templates)
6. Summary Generator
– Aggregates statuses into tibble & summary.html via render_summary
7. File System I/O
– Reads scripts, YAML; writes per-script “_log.html” and “summary.html”
8. Optional External Services
– S3 registration (import-standalone-s3-register)
– renv state capture
– User/API → Config Parser → Scheduler
– Scheduler ↔ Worker Pool
– Worker Pool → Execution Engine (spawn R sessions)
– Execution Engine → Logging Module → File System
– Execution Engine → Summary Generator → File System & return tibble
– Optional: Logging Module → External Service (S3)
c. Architectural patterns / design principles
– Modular R package: each concern in its own R/ file
– Worker pool concurrency (parallel processing)
– Pipeline pattern: Config → Schedule → Execute → Log → Summarize
– Template-driven report generation (Quarto/QMD)
– Extensibility via YAML config and options hooks
d. Key technologies & libraries
– R (base, parallel)
– yaml (config parsing)
– mdformats, quarto (HTML rendering)
– testthat (testing)
– renv (environment reproducibility)
– Continuous Integration: GitHub Actions