# novonordisk-opensource/whirl architecture

> Architecture diagram of the GitHub repository novonordisk-opensource/whirl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/novonordisk-opensource/whirl
- Repository: https://github.com/novonordisk-opensource/whirl

## Overview

##### Project type and purpose
- This is an R package (“whirl”) whose purpose is to batch-execute R (and Python) scripts under isolated sessions, capture their status/environment/logs, and render per-script and summary HTML reports. It’s a command-line / library tool, not a web app or microservice.

##### File structure
- Top level
- DESCRIPTION, NAMESPACE, R/, inst/, man/, tests/, vignettes/, .github/
- R/: all core code broken into many R modules (run.R, internal_run.R, custom_logging.R, utils.R, queue management, session setup, mdformats/quarto wrappers, etc.)
- inst/: shipped assets – QMD templates, example YAML configs, Python helper modules, RStudio addins.
- man/: .Rd docs per public function.
- tests/: testthat tests and snapshot files drive CI.
- vignettes/: user guides (quarto/Rmd).
- .github/: CI workflows for build & test.

##### README for additional insights
- Main entrypoint: run(input, n_workers, …)
- Supports either script paths or a YAML “steps” config.
- Spins up parallel workers, executes scripts in fresh R sessions, logs code, status, env info, and turns logs into HTML via mdformats/quarto.
- Produces per-script “_log.html” and an aggregated “summary.html.”
- Offers customizable hooks via whirl-options and renv integration.

1. User/API Layer
– R console CLI or RStudio addin calling whirl::run()
2. Config Parser
– Reads script paths or YAML steps file (inst/use_whirl/_whirl.yml patterns)
3. Scheduler / Worker Pool
– Manages n_workers, enqueues jobs, dequeues when workers available
4. Execution Engine
– For each script:
- Launches isolated R session (Rscript) via whirl_r_session
- Passes script to internal_run
5. Logging Module
– custom_logging captures stdout, warnings, errors, session info
– mdformats & quarto wrappers render HTML (inst/documents templates)
6. Summary Generator
– Aggregates statuses into tibble & summary.html via render_summary
7. File System I/O
– Reads scripts, YAML; writes per-script “_log.html” and “summary.html”
8. Optional External Services
– S3 registration (import-standalone-s3-register)
– renv state capture

– User/API → Config Parser → Scheduler
– Scheduler ↔ Worker Pool
– Worker Pool → Execution Engine (spawn R sessions)
– Execution Engine → Logging Module → File System
– Execution Engine → Summary Generator → File System & return tibble
– Optional: Logging Module → External Service (S3)

c. Architectural patterns / design principles
– Modular R package: each concern in its own R/ file
– Worker pool concurrency (parallel processing)
– Pipeline pattern: Config → Schedule → Execute → Log → Summarize
– Template-driven report generation (Quarto/QMD)
– Extensibility via YAML config and options hooks

d. Key technologies & libraries
– R (base, parallel)
– yaml (config parsing)
– mdformats, quarto (HTML rendering)
– testthat (testing)
– renv (environment reproducibility)
– Continuous Integration: GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User/API Layer"
        API1["run() Entry (R/run.R)"]:::compute
        API2["use_whirl Addin (R/use_whirl.R)"]:::compute
        API3["RStudio Addin Config (inst/rstudio/addins.dcf)"]:::compute
    end

    subgraph "Config Parser"
        CP1["enrich_input (R/enrich_input.R)"]:::config
        CP2["read_glob (R/read_glob.R)"]:::config
        CP3["_whirl.yml Template (inst/use_whirl/_whirl.yml)"]:::config
        CP4["_whirl.yaml Example (inst/examples/_whirl.yaml)"]:::config
    end

    subgraph "Scheduler / Worker Pool"
        SCH1["Scheduler/Queue (R/whirl_queue.R)"]:::compute
        SCH2["Queue Summary Util (R/util_queue_summary.R)"]:::compute
    end

    subgraph "Execution Engine"
        EE1["whirl_r_session (R/whirl_r_session.R)"]:::compute
        EE2["internal_run (R/internal_run.R)"]:::compute
        EE3["run_current_script (R/run_current_script.R)"]:::compute
        EE4["session (R/session.R)"]:::compute
        EE5["pb_script (R/pb_script.R)"]:::compute
        EE6["strace (R/strace.R)"]:::compute
    end

    subgraph "Logging Module"
        LM1["custom_logging (R/custom_logging.R)"]:::report
        LM2["mdformats (R/mdformats.R)"]:::report
        LM3["quarto (R/quarto.R)"]:::report
        LM4["log.qmd (inst/documents/log.qmd)"]:::report
        LM5["_quarto.yml (inst/documents/_quarto.yml)"]:::report
        LM6["python_modules.py (inst/documents/python_modules.py)"]:::report
    end

    subgraph "Summary Generator"
        SG1["render_summary (R/render_summary.R)"]:::report
        SG2["status (R/status.R)"]:::report
        SG3["summary.qmd (inst/documents/summary.qmd)"]:::report
    end

    subgraph "File System I/O"
        FS1["utils (R/utils.R)"]:::io
        FS2["normalize_with_base (R/normalize_with_base.R)"]:::io
        FS3["read_glob (R/read_glob.R)"]:::io
        FS4["render_summary (R/render_summary.R)"]:::io
        FS5["custom_logging (R/custom_logging.R)"]:::io
    end

    subgraph "Optional External Services"
        OE1["S3 Register (R/import-standalone-s3-register.R)"]:::external
        OE2["renv (R/renv.R)"]:::external
    end

    subgraph "Configuration & Hooks"
        CH1["whirl-options (R/whirl-options.R)"]:::config
        CH2["zzz init (R/zzz.R)"]:::config
    end

    subgraph "Testing & CI"
        T1["CI Workflow (.github/workflows/check_and_co.yaml)"]:::ci
        T2["Tests (tests/testthat)"]:::ci
    end

    API1 --> CP1
    API2 --> CP1
    API3 --> CP1
    CP1 --> SCH1
    SCH1 --> EE1
    SCH1 --> EE2
    EE1 --> LM1
    EE2 --> LM1
    EE3 --> LM1
    EE4 --> LM1
    EE5 --> LM1
    EE6 --> LM1
    LM1 --> FS5
    LM2 --> FS5
    LM3 --> FS5
    LM4 --> FS5
    LM5 --> FS5
    LM6 --> FS5
    EE1 --> SG1
    EE2 --> SG1
    SG1 --> FS4
    SG2 --> FS4
    SG3 --> FS4
    LM1 --> OE1
    CH1 --> EE2
    T1 --> CP1
    T1 --> SCH1
    T1 --> EE1
    T1 --> LM1
    T1 --> SG1
    T2 --> CP1
    T2 --> SCH1
    T2 --> EE1
    T2 --> LM1
    T2 --> SG1

    classDef compute fill:#D0E1FD,stroke:#0366D6,color:#000;
    classDef io fill:#DFF5D0,stroke:#228B22,color:#000;
    classDef config fill:#FFF4C2,stroke:#DAA520,color:#000;
    classDef report fill:#FFE4B5,stroke:#FF8C00,color:#000;
    classDef external fill:#E8E8E8,stroke:#696969,color:#000;
    classDef ci fill:#FDDDE6,stroke:#D73A49,color:#000;

    click API1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/run.R"
    click API2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/use_whirl.R"
    click API3 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/rstudio/addins.dcf"
    click CP1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/enrich_input.R"
    click CP2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/read_glob.R"
    click CP3 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/use_whirl/_whirl.yml"
    click CP4 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/examples/_whirl.yaml"
    click SCH1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/whirl_queue.R"
    click SCH2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/util_queue_summary.R"
    click EE1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/whirl_r_session.R"
    click EE2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/internal_run.R"
    click EE3 "https://github.com/novonordisk-opensource/whirl/blob/main/R/run_current_script.R"
    click EE4 "https://github.com/novonordisk-opensource/whirl/blob/main/R/session.R"
    click EE5 "https://github.com/novonordisk-opensource/whirl/blob/main/R/pb_script.R"
    click EE6 "https://github.com/novonordisk-opensource/whirl/blob/main/R/strace.R"
    click LM1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/custom_logging.R"
    click LM2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/mdformats.R"
    click LM3 "https://github.com/novonordisk-opensource/whirl/blob/main/R/quarto.R"
    click LM4 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/documents/log.qmd"
    click LM5 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/documents/_quarto.yml"
    click LM6 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/documents/python_modules.py"
    click SG1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/render_summary.R"
    click SG2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/status.R"
    click SG3 "https://github.com/novonordisk-opensource/whirl/blob/main/inst/documents/summary.qmd"
    click FS1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/utils.R"
    click FS2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/normalize_with_base.R"
    click FS3 "https://github.com/novonordisk-opensource/whirl/blob/main/R/read_glob.R"
    click FS4 "https://github.com/novonordisk-opensource/whirl/blob/main/R/render_summary.R"
    click FS5 "https://github.com/novonordisk-opensource/whirl/blob/main/R/custom_logging.R"
    click OE1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/import-standalone-s3-register.R"
    click OE2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/renv.R"
    click CH1 "https://github.com/novonordisk-opensource/whirl/blob/main/R/whirl-options.R"
    click CH2 "https://github.com/novonordisk-opensource/whirl/blob/main/R/zzz.R"
    click T1 "https://github.com/novonordisk-opensource/whirl/blob/main/.github/workflows/check_and_co.yaml"
    click T2 "https://github.com/novonordisk-opensource/whirl/tree/main/tests/testthat"
```
