How dune-daq/drunc works
1. Overall shape
- drunc is a Python-based run-control framework for DUNE DAQ.
- Its architecture is split into a controller plane, a process-management plane, a session-management layer, and a set of shared runtime services like FSM, broadcasting, authorization, and connectivity helpers.
- The system is highly configuration-driven: runtime behavior is defined by config files and FSM/action definitions rather than hard-coded application logic.
2. Primary user entry points
- The main executable apps live under src/drunc/apps/.
- Key front ends are controller.py, pm.py, session_manager.py, and unified_shell.py.
- There are also admin/utility tools for SSH config validation/diagnostics and a fake_daqapp_rest.py for testing REST-based child interactions.
3. Controller subsystem
- src/drunc/controller/ is the orchestration core.
- It manages a tree of child nodes through children_interface/, with support for both gRPC children and REST API children.
- The controller owns stateful behavior (stateful_node.py) and exposes an interface layer for commands, shell integration, and driver logic.
- This is the main place where run-control commands are translated into coordinated actions on downstream nodes.
4. Process manager subsystem
- src/drunc/process_manager/ is responsible for starting, stopping, and supervising DAQ processes.
- It supports multiple backends: SSH-based execution and Kubernetes-based execution.
- The interface layer mirrors the controller’s structure: CLI/shell command objects, context, and a driver-facing implementation.
- This subsystem is the boundary between run-control intent and actual process lifecycle management.
5. Session management
- src/drunc/session_manager/ tracks and coordinates run sessions.
- It provides its own service, driver, and interface layer, indicating it is an independently addressable control service rather than just an internal helper.
- In the architecture, session management sits alongside controller and process manager as a first-class operational service.
6. Finite state machine as the run-control model
- src/drunc/fsm/ encodes the operational lifecycle of the DAQ.
- It contains the FSM core, transitions, action factory, and a collection of actions that plug in run-specific behavior.
- Actions include run registry and logbook integrations, thread pinning, timing/trigger-related behavior, and user-provided run-number sources.
- This FSM layer is the canonical source of truth for allowed state transitions and run phases.
7. Broadcast/notification path
- src/drunc/broadcast/ provides a separate publish/notify path from command execution.
- It has both server-side senders and client-side handlers, with support for gRPC and Kafka-style stdout broadcast handlers.
- This is the outward-facing feedback channel used to propagate status and events to operators or monitoring consumers.
8. Security and access control
- src/drunc/authoriser/ implements authorization hooks and decorators.
- The presence of a dummy authorizer suggests the authorization layer is pluggable and can be swapped or disabled in some deployments.
- In the architecture diagram, this sits as a cross-cutting gate in front of command endpoints rather than as a standalone service.
9. Connectivity and child-process plumbing
- src/drunc/connectivity_service/ and src/drunc/processes/ handle lower-level connectivity and process lifetime concerns.
- These packages support the controller/process-manager interaction with managed children and SSH-launched processes.
- They form the glue between higher-level orchestration and the actual transport/process mechanics.
10. Protocols and service boundaries
- The repository is centered on gRPC for service communication, with some REST support for child nodes and test doubles.
- Shell-facing interfaces exist for controller, process manager, and unified shell, which implies the operator workflow is interactive as well as service-driven.
- src/drunc/grpc_testing_tools/ includes test servers, proto definitions, and connection helpers, reinforcing that gRPC is a core architectural boundary.
11. Configuration and deployment inputs
- The repo contains many config artifacts under src/drunc/data/ and config/tests/.
- There are JSON schemas and canned deployment configs for SSH and K8s process-manager setups, plus XML configs used in tests.
- This suggests deployments are assembled from config rather than compiled into a fixed topology.
12. Support tooling and verification
- The extensive tests/ tree mirrors the main subsystems almost one-for-one, which shows the architecture is decomposed into independently testable modules.
- GitHub Actions cover linting, pytest, docs checks, and docs deployment.
- Documentation is a first-class part of the repo structure, with subsystem-specific docs for controller, process manager, FSM, messaging format, and unified shell.
Main entry points: controller.py Controller core · process_manager.py PM core · unified_shell.py Unified shell · session_manager.py Session core
Components 23
Entry points
Operator-facing executables and shell front ends.
- Controller app · Main controller executable.
src/drunc/apps/controller.py - PM app · Process-manager executable.
src/drunc/apps/pm.py - Session app · Session-manager executable.
src/drunc/apps/session_manager.py - Unified shell · Operator shell combining major services.
src/drunc/apps/unified_shell.py
Controller plane
Orchestrates child nodes and command execution.
- Controller core · Coordinates commands and downstream state.
src/drunc/controller/controller.py - State node · Holds controller state and transitions.
src/drunc/controller/stateful_node.py - gRPC child · Child-node adapter over gRPC.
src/drunc/controller/children_interface/grpc_child.py - REST child · Child-node adapter over REST.
src/drunc/controller/children_interface/rest_api_child.py
Process manager
Starts, stops, and supervises DAQ processes.
- PM core · Coordinates process lifecycle operations.
src/drunc/process_manager/process_manager.py - SSH backend · Launches and supervises processes over SSH.
src/drunc/process_manager/ssh_process_manager.py - K8s backend · Launches and supervises processes in Kubernetes.
src/drunc/process_manager/k8s_process_manager.py
Session management
Tracks and coordinates operational sessions.
- Session core · Owns session lifecycle and coordination.
src/drunc/session_manager/session_manager.py - Session API · Session-manager interface layer.
src/drunc/session_manager/interface/session_manager.py
FSM runtime
Defines lifecycle states, transitions, and actions.
- FSM core · Defines legal run-control states.
src/drunc/fsm/core.py - Transitions · Encodes allowed lifecycle transitions.
src/drunc/fsm/transition.py - Action factory · Instantiates FSM actions from configuration.
src/drunc/fsm/action_factory.py - Run registry · Example action for run registration.
src/drunc/fsm/actions/db_run_registry.py
Broadcasting
Notification and status dissemination channels.
- Broadcast server · Publishes runtime events to clients.
src/drunc/broadcast/server/grpc_servicer.py - Broadcast client · Consumes broadcast events on the client side.
src/drunc/broadcast/client/broadcast_handler.py
Security
Authorization hooks and policy plumbing.
- Auth decorators · Authorization hooks around protected operations.
src/drunc/authoriser/decorators.py
Plumbing
Connectivity and process lifetime support.
- Connectivity client · Connectivity helper for controller/process links.
src/drunc/connectivity_service/client.py - Process lifetime · Manages SSH-launched process lifetime.
src/drunc/processes/ssh_process_lifetime_manager.py
gRPC tools
Service testing and connection scaffolding.
- Test proto · Shared test service definitions and stubs.
src/drunc/grpc_testing_tools/test_services.proto
Connections 22
- Controller app → Controller core: starts · Controller executable boots the orchestration core.
- PM app → PM core: starts · Process-manager executable boots the lifecycle service.
- Session app → Session core: starts · Session executable boots the session service.
- Unified shell → Controller core: connects · Unified shell can target controller operations.
- Unified shell → PM core: connects · Unified shell can target process-manager operations.
- Unified shell → Session core: connects · Unified shell can target session operations.
- Controller core → State node: tracks state · Controller behavior is anchored in node state.
- Controller core → gRPC child: drives · Controller issues commands to gRPC children.
- Controller core → REST child: drives · Controller issues commands to REST children.
- Controller core → Auth decorators: gated by · Controller endpoints are authorization-protected.
- Controller core → Connectivity client: uses · Controller relies on connectivity helpers.
- PM core → SSH backend: backend · SSH is one process-management implementation.
- PM core → K8s backend: backend · Kubernetes is another process-management implementation.
- PM core → Process lifetime: uses · Process manager delegates lifecycle mechanics.
- PM core → Auth decorators: gated by · Process-manager endpoints are authorization-protected.
- Session core → Session API: exposes · Session core is surfaced through an interface layer.
- FSM core → Transitions: defines · FSM core depends on transition definitions.
- FSM core → Action factory: invokes · State changes trigger configured actions.
- Action factory → Run registry: creates · Representative run-specific action is wired in.
- Broadcast server → Broadcast client: delivers to · Server-side broadcast is consumed by clients.
- Test proto → Controller core: tests · gRPC test services exercise controller-style boundaries.
- Test proto → PM core: tests · gRPC test services exercise process-manager boundaries.