belle2/b2luigi

1. Overall shape

- This is a Python library, not an application server: b2luigi extends Luigi with Belle II–oriented execution helpers.

- The main architecture is a layered task system: user-defined tasks sit on top of a core Luigi-compatible abstraction, with optional batch execution and remote I/O backends underneath.

2. Core orchestration layer

- b2luigi/core is the heart of the package.

- It defines the task model (task.py), parameters (parameter.py), targets (target.py), executable/task dispatch support (executable.py, dispatchable_task.py), temporary-output handling (temporary_wrapper.py), and runtime settings (settings.py).

- This layer is where Luigi concepts are adapted and wrapped so tasks can run locally or be handed off to batch systems.

3. Dependency and execution flow

- The normal flow is: a user defines a task → declares parameters, requirements, and outputs → Luigi resolves the dependency graph → b2luigi either runs the task locally or submits it to a batch backend.

- Outputs are represented as targets, so task completion is determined by target existence rather than by imperative status tracking.

4. Remote storage boundary

- b2luigi/core/remote_target contains storage adapters for WebDAV and XRootD.

- These are the main boundary between compute and persistent data: tasks can write to or verify outputs on remote storage rather than only on the local filesystem.

- Tests for WebDAV/XRootD indicate these are first-class storage backends, not incidental utilities.

5. Batch orchestration layer

- b2luigi/batch handles scheduling and execution outside the local process.

- workers.py and cache.py suggest a small runtime/control plane for worker state and job metadata.

- This layer abstracts the common pattern of “prepare job description, submit, poll status, map scheduler state back to Luigi task state.”

6. Scheduler-specific process adapters

- b2luigi/batch/processes contains backend implementations for Apptainer, HTCondor, LSF, Slurm, and gbasf2.

- Each adapter is responsible for translating a Luigi task into the scheduler’s submission format and lifecycle semantics.

- The templates/ directory shows that at least some backends generate wrapper scripts from templates rather than hardcoding job payloads.

7. Specialized Belle II / gbasf2 support

- b2luigi/batch/processes/gbasf2* is a more specialized subsystem for the Belle II grid workflow.

- The helper modules cover job status inspection, dataset listing, proxy handling, downloads, and pickle serialization.

- This implies a separate operational path from standard HPC schedulers: it integrates with external grid services and their job/state conventions.

8. Containerized execution

- apptainer.py indicates support for running tasks inside containers.

- Architecturally, this sits between task submission and actual task execution, providing environment isolation when the host system cannot guarantee dependencies.

9. CLI boundary

- b2luigi/cli provides the user entry point: argument parsing, process management, and a runner.

- The CLI is the thin shell around the orchestration layer; it likely constructs the runtime configuration, instantiates the scheduler/process, and launches the task graph.

- Tests around dry-run and extra argument handling suggest the CLI also mediates what belongs to Luigi/b2luigi versus user-defined script arguments.

10. Basf2 integration layer

- b2luigi/basf2_helper adds task and target helpers for basf2, the Belle II analysis framework.

- This is a domain-specific extension layer that bridges the generic Luigi/batch machinery with Belle II analysis workflows and file conventions.

11. Configuration and runtime control

- settings.py, example settings.json files, and the examples/*/setup_script.sh files show that runtime behavior is configured externally per environment or backend.

- In practice, backend choice, execution mode, and environment setup are controlled by configuration plus the launcher script, not hardwired into task code.

12. Documentation and examples as architecture guides

- The docs/ and examples/ trees are substantial and reflect the intended architecture: tutorial tasks, batch examples, scheduler queries, remote target usage, and basf2 workflows.

- The examples demonstrate the supported deployment modes: local Luigi execution, batch submission, grid workflows, and remote storage-backed analyses.

13. Test structure mirrors the subsystem boundaries

- tests/core, tests/batch, tests/cli, and tests/basf2_helper map directly onto the main layers.

- This suggests the package is designed around clear contracts between task abstraction, execution backend, CLI front end, and external service integrations.

14. What an architecture diagram should emphasize

- Show b2luigi as a wrapper/extension around Luigi.

- Put the core task graph in the center.

- Branch from it to: local execution, batch backends, remote storage targets, and basf2-specific helpers.

- Treat HTCondor/LSF/Slurm/gbasf2/Apptainer as interchangeable execution adapters, with WebDAV/XRootD as storage adapters and the CLI as the user entry point.