# belle2/b2luigi architecture

> Architecture diagram of the GitHub repository belle2/b2luigi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/belle2/b2luigi
- Repository: https://github.com/belle2/b2luigi

## Overview

1. **Overall shape**
   - This is a Python library, not an application server: `b2luigi` extends **Luigi** with Belle II–oriented execution helpers.
   - The main architecture is a layered task system: user-defined tasks sit on top of a core Luigi-compatible abstraction, with optional batch execution and remote I/O backends underneath.

2. **Core orchestration layer**
   - `b2luigi/core` is the heart of the package.
   - It defines the task model (`task.py`), parameters (`parameter.py`), targets (`target.py`), executable/task dispatch support (`executable.py`, `dispatchable_task.py`), temporary-output handling (`temporary_wrapper.py`), and runtime settings (`settings.py`).
   - This layer is where Luigi concepts are adapted and wrapped so tasks can run locally or be handed off to batch systems.

3. **Dependency and execution flow**
   - The normal flow is: a user defines a task → declares parameters, requirements, and outputs → Luigi resolves the dependency graph → `b2luigi` either runs the task locally or submits it to a batch backend.
   - Outputs are represented as targets, so task completion is determined by target existence rather than by imperative status tracking.

4. **Remote storage boundary**
   - `b2luigi/core/remote_target` contains storage adapters for **WebDAV** and **XRootD**.
   - These are the main boundary between compute and persistent data: tasks can write to or verify outputs on remote storage rather than only on the local filesystem.
   - Tests for WebDAV/XRootD indicate these are first-class storage backends, not incidental utilities.

5. **Batch orchestration layer**
   - `b2luigi/batch` handles scheduling and execution outside the local process.
   - `workers.py` and `cache.py` suggest a small runtime/control plane for worker state and job metadata.
   - This layer abstracts the common pattern of “prepare job description, submit, poll status, map scheduler state back to Luigi task state.”

6. **Scheduler-specific process adapters**
   - `b2luigi/batch/processes` contains backend implementations for **Apptainer**, **HTCondor**, **LSF**, **Slurm**, and **gbasf2**.
   - Each adapter is responsible for translating a Luigi task into the scheduler’s submission format and lifecycle semantics.
   - The `templates/` directory shows that at least some backends generate wrapper scripts from templates rather than hardcoding job payloads.

7. **Specialized Belle II / gbasf2 support**
   - `b2luigi/batch/processes/gbasf2*` is a more specialized subsystem for the Belle II grid workflow.
   - The helper modules cover job status inspection, dataset listing, proxy handling, downloads, and pickle serialization.
   - This implies a separate operational path from standard HPC schedulers: it integrates with external grid services and their job/state conventions.

8. **Containerized execution**
   - `apptainer.py` indicates support for running tasks inside containers.
   - Architecturally, this sits between task submission and actual task execution, providing environment isolation when the host system cannot guarantee dependencies.

9. **CLI boundary**
   - `b2luigi/cli` provides the user entry point: argument parsing, process management, and a runner.
   - The CLI is the thin shell around the orchestration layer; it likely constructs the runtime configuration, instantiates the scheduler/process, and launches the task graph.
   - Tests around dry-run and extra argument handling suggest the CLI also mediates what belongs to Luigi/b2luigi versus user-defined script arguments.

10. **Basf2 integration layer**
   - `b2luigi/basf2_helper` adds task and target helpers for **basf2**, the Belle II analysis framework.
   - This is a domain-specific extension layer that bridges the generic Luigi/batch machinery with Belle II analysis workflows and file conventions.

11. **Configuration and runtime control**
   - `settings.py`, example `settings.json` files, and the `examples/*/setup_script.sh` files show that runtime behavior is configured externally per environment or backend.
   - In practice, backend choice, execution mode, and environment setup are controlled by configuration plus the launcher script, not hardwired into task code.

12. **Documentation and examples as architecture guides**
   - The `docs/` and `examples/` trees are substantial and reflect the intended architecture: tutorial tasks, batch examples, scheduler queries, remote target usage, and basf2 workflows.
   - The examples demonstrate the supported deployment modes: local Luigi execution, batch submission, grid workflows, and remote storage-backed analyses.

13. **Test structure mirrors the subsystem boundaries**
   - `tests/core`, `tests/batch`, `tests/cli`, and `tests/basf2_helper` map directly onto the main layers.
   - This suggests the package is designed around clear contracts between task abstraction, execution backend, CLI front end, and external service integrations.

14. **What an architecture diagram should emphasize**
   - Show `b2luigi` as a wrapper/extension around Luigi.
   - Put the core task graph in the center.
   - Branch from it to: local execution, batch backends, remote storage targets, and basf2-specific helpers.
   - Treat HTCondor/LSF/Slurm/gbasf2/Apptainer as interchangeable execution adapters, with WebDAV/XRootD as storage adapters and the CLI as the user entry point.

Main entry points: [`b2luigi/core/task.py`](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/task.py) (Task model), [`b2luigi/batch/processes`](https://github.com/belle2/b2luigi/tree/main/b2luigi/batch/processes) (Adapters), [`b2luigi/basf2_helper`](https://github.com/belle2/b2luigi/tree/main/b2luigi/basf2_helper) (basf2 helper), [`b2luigi/batch`](https://github.com/belle2/b2luigi/tree/main/b2luigi/batch) (Batch core)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core"]
  node_core_task["Task model<br/>core task<br/>[task.py]"]
  node_core_param["Parameters<br/>core params<br/>[parameter.py]"]
  node_core_target["Targets<br/>core outputs<br/>[target.py]"]
  node_core_exec["Dispatch<br/>core runtime"]
  node_core_temp["Temp wrapper<br/>core temp"]
  node_core_settings["Settings<br/>core config<br/>[settings.py]"]
end

subgraph group_batch["Batch"]
  node_batch_core["Batch core<br/>batch runtime"]
  node_batch_workers["Workers<br/>batch control<br/>[workers.py]"]
  node_batch_processes["Adapters<br/>batch backends"]
  node_apptainer["Apptainer<br/>container adapter<br/>[apptainer.py]"]
  node_htcondor["HTCondor<br/>scheduler adapter<br/>[htcondor.py]"]
  node_lsf["LSF<br/>scheduler adapter<br/>[lsf.py]"]
  node_slurm["Slurm<br/>scheduler adapter<br/>[slurm.py]"]
  node_gbasf2["gbasf2<br/>grid adapter<br/>[gbasf2.py]"]
end

subgraph group_cli["CLI"]
  node_cli_runner["Runner<br/>cli launcher<br/>[runner.py]"]
  node_cli_args["Arguments<br/>cli parsing<br/>[arguments.py]"]
end

subgraph group_basf2["basf2"]
  node_basf2_helper["basf2 helper<br/>domain bridge"]
end

subgraph group_storage["Remote I/O"]
  node_remote_target[("Remote targets<br/>storage adapters")]
  node_webdav["WebDAV<br/>storage backend<br/>[webdav.py]"]
  node_xrootd["XRootD<br/>storage backend<br/>[xrootd.py]"]
end

node_luigi(("Luigi<br/>external scheduler"))

node_luigi -->|"extends"| node_core_task
node_core_param -->|"configures"| node_core_task
node_core_target -->|"defines done-ness"| node_core_task
node_core_exec -->|"runs"| node_core_task
node_core_temp -->|"wraps"| node_core_task
node_core_settings -->|"controls"| node_core_task
node_core_target -->|"uses"| node_remote_target
node_remote_target -->|"adapter"| node_webdav
node_remote_target -->|"adapter"| node_xrootd
node_core_task -->|"submits to"| node_batch_core
node_batch_core -->|"tracks"| node_batch_workers
node_batch_core -->|"delegates"| node_batch_processes
node_batch_processes -->|"backend"| node_apptainer
node_batch_processes -->|"backend"| node_htcondor
node_batch_processes -->|"backend"| node_lsf
node_batch_processes -->|"backend"| node_slurm
node_batch_processes -->|"backend"| node_gbasf2
node_cli_args -->|"drives"| node_cli_runner
node_cli_runner -->|"launches"| node_core_task
node_basf2_helper -->|"specializes"| node_core_task
node_basf2_helper -->|"specializes"| node_core_target
node_basf2_helper -->|"uses"| node_batch_processes

click node_core_task "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/task.py"
click node_core_param "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/parameter.py"
click node_core_target "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/target.py"
click node_core_exec "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/dispatchable_task.py"
click node_core_temp "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/temporary_wrapper.py"
click node_core_settings "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/settings.py"
click node_remote_target "https://github.com/belle2/b2luigi/tree/main/b2luigi/core/remote_target"
click node_webdav "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/remote_target/webdav.py"
click node_xrootd "https://github.com/belle2/b2luigi/blob/main/b2luigi/core/remote_target/xrootd.py"
click node_batch_core "https://github.com/belle2/b2luigi/tree/main/b2luigi/batch"
click node_batch_workers "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/workers.py"
click node_batch_processes "https://github.com/belle2/b2luigi/tree/main/b2luigi/batch/processes"
click node_apptainer "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/apptainer.py"
click node_htcondor "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/htcondor.py"
click node_lsf "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/lsf.py"
click node_slurm "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/slurm.py"
click node_gbasf2 "https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/gbasf2.py"
click node_cli_runner "https://github.com/belle2/b2luigi/blob/main/b2luigi/cli/runner.py"
click node_cli_args "https://github.com/belle2/b2luigi/blob/main/b2luigi/cli/arguments.py"
click node_basf2_helper "https://github.com/belle2/b2luigi/tree/main/b2luigi/basf2_helper"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_core_task,node_core_param,node_core_target,node_core_exec,node_core_temp,node_core_settings toneBlue
class node_batch_core,node_batch_workers,node_batch_processes,node_apptainer,node_htcondor,node_lsf,node_slurm,node_gbasf2 toneAmber
class node_cli_runner,node_cli_args toneMint
class node_basf2_helper toneRose
class node_remote_target,node_webdav,node_xrootd toneIndigo
class node_luigi toneNeutral
```

## Components

### Core

Luigi-compatible task model and runtime settings

- [Task model](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/task.py): core task · Luigi-compatible task abstraction and execution contract · `b2luigi/core/task.py`
- [Parameters](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/parameter.py): core params · Task parameter definitions and grouping behavior · `b2luigi/core/parameter.py`
- [Targets](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/target.py): core outputs · Completion by target existence rather than imperative status · `b2luigi/core/target.py`
- [Dispatch](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/dispatchable_task.py): core runtime · Executable and dispatchable task support · `b2luigi/core/dispatchable_task.py`
- [Temp wrapper](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/temporary_wrapper.py): core temp · Temporary output orchestration for task execution · `b2luigi/core/temporary_wrapper.py`
- [Settings](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/settings.py): core config · Runtime settings and execution controls · `b2luigi/core/settings.py`

### Batch

Submission and worker control for external execution

- [Batch core](https://github.com/belle2/b2luigi/tree/main/b2luigi/batch): batch runtime · Job orchestration, cache, and worker state · `b2luigi/batch`
- [Workers](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/workers.py): batch control · Worker state and scheduler-facing control plane · `b2luigi/batch/workers.py`
- [Adapters](https://github.com/belle2/b2luigi/tree/main/b2luigi/batch/processes): batch backends · Scheduler and container submission backends · `b2luigi/batch/processes`
- [Apptainer](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/apptainer.py): container adapter · Containerized execution adapter · `b2luigi/batch/processes/apptainer.py`
- [HTCondor](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/htcondor.py): scheduler adapter · HTCondor submission and lifecycle adapter · `b2luigi/batch/processes/htcondor.py`
- [LSF](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/lsf.py): scheduler adapter · LSF submission and lifecycle adapter · `b2luigi/batch/processes/lsf.py`
- [Slurm](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/slurm.py): scheduler adapter · Slurm submission and lifecycle adapter · `b2luigi/batch/processes/slurm.py`
- [gbasf2](https://github.com/belle2/b2luigi/blob/main/b2luigi/batch/processes/gbasf2.py): grid adapter · Belle II grid workflow adapter and helpers · `b2luigi/batch/processes/gbasf2.py`

### CLI

User entry points and process launcher

- [Runner](https://github.com/belle2/b2luigi/blob/main/b2luigi/cli/runner.py): cli launcher · Launches b2luigi runs from the command line · `b2luigi/cli/runner.py`
- [Arguments](https://github.com/belle2/b2luigi/blob/main/b2luigi/cli/arguments.py): cli parsing · Parses b2luigi and user command-line arguments · `b2luigi/cli/arguments.py`

### basf2

Belle II domain helpers

- [basf2 helper](https://github.com/belle2/b2luigi/tree/main/b2luigi/basf2_helper): domain bridge · Belle II task and target helpers · `b2luigi/basf2_helper`

### Remote I/O

Remote output and input targets

- [Remote targets](https://github.com/belle2/b2luigi/tree/main/b2luigi/core/remote_target): storage adapters · Remote storage boundary for persistent outputs · `b2luigi/core/remote_target`
- [WebDAV](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/remote_target/webdav.py): storage backend · WebDAV remote target adapter · `b2luigi/core/remote_target/webdav.py`
- [XRootD](https://github.com/belle2/b2luigi/blob/main/b2luigi/core/remote_target/xrootd.py): storage backend · XRootD remote target adapter · `b2luigi/core/remote_target/xrootd.py`

### Other components

- **Luigi**: external scheduler · Upstream workflow engine that b2luigi extends

## Connections

- Luigi → Task model: extends · b2luigi adapts Luigi task semantics
- Parameters → Task model: configures · Parameters define task inputs and grouping
- Targets → Task model: defines done-ness · Task completion is derived from targets
- Dispatch → Task model: runs · Executable task support sits under the task model
- Temp wrapper → Task model: wraps · Temporary output handling for task execution
- Settings → Task model: controls · Runtime settings shape task execution
- Targets → Remote targets: uses · Targets may be backed by remote storage
- Remote targets → WebDAV: adapter · WebDAV backend implementation
- Remote targets → XRootD: adapter · XRootD backend implementation
- Task model → Batch core: submits to · Tasks can be handed off to batch execution
- Batch core → Workers: tracks · Worker state and job metadata management
- Batch core → Adapters: delegates · Submission logic is delegated to backend adapters
- Adapters → Apptainer: backend · Container execution backend
- Adapters → HTCondor: backend · HTCondor backend
- Adapters → LSF: backend · LSF backend
- Adapters → Slurm: backend · Slurm backend
- Adapters → gbasf2: backend · Belle II grid backend
- Arguments → Runner: drives · Argument parsing feeds the runner
- Runner → Task model: launches · CLI starts task execution through the core layer
- basf2 helper → Task model: specializes · Belle II-specific task helpers build on core tasks
- basf2 helper → Targets: specializes · Belle II-specific targets integrate with core outputs
- basf2 helper → Adapters: uses · Basf2 workflows can run through batch adapters