# slaclab/pysmurf architecture

> Architecture diagram of the GitHub repository slaclab/pysmurf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/slaclab/pysmurf
- Repository: https://github.com/slaclab/pysmurf

## Overview

1. **Repository purpose**
   - This is the `pysmurf` control stack for SMuRF hardware: a Python-centric system for operating readout hardware, configuring experiments, streaming data, and running analysis/tuning workflows.
   - It is not a single app; it is a layered toolkit spanning control, hardware I/O, data processing, emulation, and operational scripts.

2. **Main runtime split: Python control layer + C++ core**
   - The user-facing package lives under `python/pysmurf`, while performance-critical streaming/packet logic is implemented in C++ under `src/smurf` with headers in `include/smurf`.
   - The Python package exposes higher-level control and analysis APIs; the C++ library handles low-level data handling, packet conversion, counters, and transmitters.

3. **Primary user-facing subsystem**
   - `python/pysmurf/client` is the operator-facing API and command layer.
   - It contains:
     - `base/` for the main control abstractions, config handling, and logging.
     - `command/` for executable-style hardware commands and operational utilities.
     - `tune/` for tuning workflows.
     - `debug/` for diagnostics and experiment-specific analysis helpers.
     - `util/` and `watchdog/` for supporting tools and health checks.
   - This is the layer most likely imported by experiment scripts and run interactively.

4. **Hardware/backend abstraction**
   - `python/pysmurf/core/roots` defines the backend entry points for different deployment modes:
     - Ethernet vs PCIe variants.
     - Dev board vs cryogenic module variants.
     - `EmulationRoot` for simulated operation.
   - This is the architectural boundary between the Python control world and the actual SMuRF hardware transport.

5. **Data acquisition and stream-processing pipeline**
   - The core streaming path is organized as:
     - device input (`PcieCard`, `UdpReceiver`, `SmurfApplication`)
     - packet/header normalization (`Header2Smurf`, `SmurfHeader`, `SmurfPacket`)
     - processing/statistics (`SmurfProcessor`, `FrameStatistics`)
     - output/transmission (`BaseTransmitter`, `DataToFile`, transmitter channels)
   - The design suggests a pipeline architecture where incoming frames are converted into a SMuRF-specific representation, analyzed, and then forwarded to sinks such as files or downstream consumers.

6. **Emulation and offline data support**
   - `python/pysmurf/core/emulators` and `src/smurf/core/emulators` provide simulated stream sources, including file-backed and synthetic data generation.
   - This allows the same processing stack to run without hardware, which is important for testing, development, and demos.
   - The presence of `StreamDataSource`, `StreamDataEmulatorI16/I32`, and `DataFromFile` indicates an explicit offline/virtual data path parallel to the live hardware path.

7. **Publisher/receiver and utility layer**
   - `python/pysmurf/core/utilities` includes support components like a data receiver, setup grouping, and a publisher.
   - These pieces look like the integration boundary for distributing processed stream data and coordinating system initialization.
   - They sit between the low-level core pipeline and higher-level client workflows.

8. **Server-side launch scripts**
   - `server_scripts/` and `python/pysmurf/core/server_scripts/` provide entry points for starting different backend configurations.
   - The top-level scripts map to the root/backend variants in the codebase, such as Ethernet/PCIe and emulation modes.
   - This indicates the server process is launched separately from the client and selected by deployment environment.

9. **Configuration is experiment/site driven**
   - `cfg_files/` contains many site- and experiment-specific `.cfg` files.
   - These are not generic defaults; they encode deployment-specific SMuRF startup and experiment settings for different labs and instruments.
   - In architecture terms, config files are a major control-plane input that parameterize the same software stack for different hardware installations.

10. **Deployment and packaging**
   - The repo is packaged as a Python distribution (`pyproject.toml`, `requirements.txt`) with a compiled C++ extension/build system (`CMakeLists.txt`, `src/`, `include/`).
   - Docker support is split into client and server images under `docker/client` and `docker/server`, reflecting the separation between operator tooling and hardware-facing services.
   - `smurfConfig.cmake.in` and `releaseGen.py` suggest release packaging and native-library build integration.

11. **Documentation and validation**
   - `docs/` contains Sphinx documentation divided into client, core, and user sections, mirroring the code architecture.
   - `tests/core/` validates key low-level processing behavior such as the processor, filter, and unwrapper logic.
   - This reinforces that the C++ core is the most algorithmically sensitive part of the system.

12. **Operational and exploratory code**
   - `scratch/` contains many one-off analysis scripts, notebooks, and experiment prototypes.
   - These are useful for users but are not core architecture; they sit outside the supported product path and should be treated as experimental tooling.

13. **Overall data/control flow**
   - Typical flow is: operator script or client API → configured root/backend → hardware or emulator input stream → C++ processing/conversion pipeline → transmitter/publisher/file output → analysis/debug/tuning tooling.
   - The main architectural boundary is between the Python control plane and the native streaming pipeline, with hardware/emulation as interchangeable sources behind the same root abstractions.

14. **What to show in an architecture diagram**
   - Show distinct boxes for: client/control API, backend root/transport selection, data acquisition/processing core, output/transmit/publish sinks, configuration inputs, and emulation/offline sources.
   - Also show deployment separation between client and server containers/processes, and the Python/C++ implementation boundary inside the core stack.

Main entry points: [`src/smurf/core/processors/SmurfProcessor.cpp`](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/processors/SmurfProcessor.cpp) (Processor), [`python/pysmurf/core/roots/Common.py`](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/roots/Common.py) (Roots), [`python/pysmurf/client/base/base_class.py`](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/base/base_class.py) (Client base), [`python/pysmurf/core/utilities/_SmurfPublisher.py`](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/utilities/_SmurfPublisher.py) (Core utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_client["Python control plane"]
  node_n_client_base["Client base<br/>Python API<br/>[base_class.py]"]
  node_n_client_command["Commands<br/>[smurf_command.py]"]
  node_n_client_tune["Tune workflows<br/>Tuning<br/>[smurf_tune.py]"]
  node_n_client_debug["Debug tools<br/>Diagnostics<br/>[smurf_noise.py]"]
  node_n_client_util["Client utils<br/>Support tools<br/>[pub.py]"]
end

subgraph group_grp_core["Streaming core"]
  node_n_core_roots["Roots<br/>Backend selection<br/>[Common.py]"]
  node_n_core_devices["Devices<br/>Hardware adapters"]
  node_n_core_converters["Converters<br/>Packet normalization<br/>[_Header2Smurf.py]"]
  node_n_core_counters["Counters<br/>Stream stats"]
  node_n_core_emulators["Emulators<br/>Offline streams"]
  node_n_core_tx["Transmitters<br/>Output sinks<br/>[_DataToFile.py]"]
  node_n_core_utils["Core utilities<br/>Coordination<br/>[_SmurfPublisher.py]"]
end

subgraph group_grp_native["Native C++ core"]
  node_n_native_common["Packet types<br/>C++ common<br/>[SmurfPacket.h]"]
  node_n_native_converters["Header convert<br/>C++ converter<br/>[Header2Smurf.cpp]"]
  node_n_native_processors["Processor<br/>C++ processing<br/>[SmurfProcessor.cpp]"]
  node_n_native_counters["Frame stats<br/>C++ counters"]
  node_n_native_emulators["Native emulators<br/>C++ sources"]
  node_n_native_tx["Native tx<br/>C++ transmitters"]
end

subgraph group_grp_ops["Deployment and support"]
  node_n_cfg["Config files<br/>Site config<br/>[template.cfg]"]
  node_n_server["Server scripts<br/>Launch entrypoints<br/>[emulate.py]"]
  node_n_docs["Docs<br/>[index.rst]"]
  node_n_tests["Core tests<br/>Validation"]
end

node_n_client_base -->|"selects backend"| node_n_core_roots
node_n_client_command -->|"uses"| node_n_client_base
node_n_client_tune -->|"drives"| node_n_client_base
node_n_client_debug -->|"analyzes"| node_n_client_base
node_n_client_util -->|"bridges"| node_n_core_utils
node_n_core_roots -->|"wires up"| node_n_core_devices
node_n_core_roots -->|"can use"| node_n_core_emulators
node_n_core_devices -->|"feeds"| node_n_core_converters
node_n_core_converters -->|"updates"| node_n_core_counters
node_n_core_converters -->|"forwards"| node_n_core_tx
node_n_core_counters -->|"publishes"| node_n_core_utils
node_n_core_emulators -->|"emulates"| node_n_core_devices
node_n_core_tx -->|"delivers"| node_n_core_utils
node_n_core_utils -->|"supports"| node_n_client_debug
node_n_core_roots -->|"launched by"| node_n_server
node_n_cfg -->|"parameterizes"| node_n_core_roots
node_n_server -->|"loads"| node_n_cfg
node_n_native_common -->|"defines"| node_n_native_converters
node_n_native_common -->|"defines"| node_n_native_processors
node_n_native_converters -->|"prepares"| node_n_native_processors
node_n_native_processors -->|"updates"| node_n_native_counters
node_n_native_emulators -->|"feeds"| node_n_native_processors
node_n_native_processors -->|"outputs"| node_n_native_tx
node_n_tests -->|"validates"| node_n_native_processors
node_n_tests -->|"validates"| node_n_native_tx
node_n_docs -->|"documents"| node_n_client_base
node_n_docs -->|"documents"| node_n_core_roots
node_n_docs -->|"documents"| node_n_native_processors

click node_n_client_base "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/base/base_class.py"
click node_n_client_command "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/command/smurf_command.py"
click node_n_client_tune "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/tune/smurf_tune.py"
click node_n_client_debug "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/debug/smurf_noise.py"
click node_n_client_util "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/util/pub.py"
click node_n_core_roots "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/roots/Common.py"
click node_n_core_devices "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/devices/_SmurfApplication.py"
click node_n_core_converters "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/conventers/_Header2Smurf.py"
click node_n_core_counters "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/counters/_FrameStatistics.py"
click node_n_core_emulators "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/emulators/_StreamDataSource.py"
click node_n_core_tx "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/transmitters/_DataToFile.py"
click node_n_core_utils "https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/utilities/_SmurfPublisher.py"
click node_n_native_common "https://github.com/slaclab/pysmurf/blob/main/include/smurf/core/common/SmurfPacket.h"
click node_n_native_converters "https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/conventers/Header2Smurf.cpp"
click node_n_native_processors "https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/processors/SmurfProcessor.cpp"
click node_n_native_counters "https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/counters/FrameStatistics.cpp"
click node_n_native_emulators "https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/emulators/StreamDataEmulator.cpp"
click node_n_native_tx "https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/transmitters/BaseTransmitter.cpp"
click node_n_cfg "https://github.com/slaclab/pysmurf/blob/main/cfg_files/template/template.cfg"
click node_n_server "https://github.com/slaclab/pysmurf/blob/main/server_scripts/emulate.py"
click node_n_docs "https://github.com/slaclab/pysmurf/blob/main/docs/index.rst"
click node_n_tests "https://github.com/slaclab/pysmurf/blob/main/tests/core/validate_base_tx.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_client_base,node_n_client_command,node_n_client_tune,node_n_client_debug,node_n_client_util toneBlue
class node_n_core_roots,node_n_core_devices,node_n_core_converters,node_n_core_counters,node_n_core_emulators,node_n_core_tx,node_n_core_utils toneAmber
class node_n_native_common,node_n_native_converters,node_n_native_processors,node_n_native_counters,node_n_native_emulators,node_n_native_tx toneMint
class node_n_cfg,node_n_server,node_n_docs,node_n_tests toneRose
```

## Components

### Python control plane

Operator-facing APIs, commands, tuning, diagnostics, and utilities.

- [Client base](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/base/base_class.py): Python API · Main control abstractions, config, logging, and control state. · `python/pysmurf/client/base/base_class.py`
- [Commands](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/command/smurf_command.py): Operator commands · Executable-style hardware commands and operational utilities. · `python/pysmurf/client/command/smurf_command.py`
- [Tune workflows](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/tune/smurf_tune.py): Tuning · Tuning and calibration workflows used during operation. · `python/pysmurf/client/tune/smurf_tune.py`
- [Debug tools](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/debug/smurf_noise.py): Diagnostics · Experiment-specific analysis and diagnostic helpers. · `python/pysmurf/client/debug/smurf_noise.py`
- [Client utils](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/client/util/pub.py): Support tools · File reading, listening, publishing, and small operational helpers. · `python/pysmurf/client/util/pub.py`

### Streaming core

Backend roots, device adapters, emulation, processing, and sinks.

- [Roots](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/roots/Common.py): Backend selection · Entry points for Ethernet, PCIe, dev-board, cryogenic, and emulated deployments. · `python/pysmurf/core/roots/Common.py`
- [Devices](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/devices/_SmurfApplication.py): Hardware adapters · Python-facing adapters for card, UDP receiver, application, and processor objects. · `python/pysmurf/core/devices/_SmurfApplication.py`
- [Converters](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/conventers/_Header2Smurf.py): Packet normalization · Transforms device headers and frames into SMuRF-specific stream objects. · `python/pysmurf/core/conventers/_Header2Smurf.py`
- [Counters](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/counters/_FrameStatistics.py): Stream stats · Frame statistics and stream counters used for monitoring and validation. · `python/pysmurf/core/counters/_FrameStatistics.py`
- [Emulators](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/emulators/_StreamDataSource.py): Offline streams · Synthetic and file-backed data sources for simulated operation. · `python/pysmurf/core/emulators/_StreamDataSource.py`
- [Transmitters](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/transmitters/_DataToFile.py): Output sinks · Python-side transmitters including file output and publisher integration. · `python/pysmurf/core/transmitters/_DataToFile.py`
- [Core utilities](https://github.com/slaclab/pysmurf/blob/main/python/pysmurf/core/utilities/_SmurfPublisher.py): Coordination · Setup groups, data receiver, and publisher coordination utilities. · `python/pysmurf/core/utilities/_SmurfPublisher.py`

### Native C++ core

Performance-critical packet, processing, and transmitter implementation.

- [Packet types](https://github.com/slaclab/pysmurf/blob/main/include/smurf/core/common/SmurfPacket.h): C++ common · Smurf headers, packets, timing, and shared native data structures. · `include/smurf/core/common/SmurfPacket.h`
- [Header convert](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/conventers/Header2Smurf.cpp): C++ converter · Native header-to-SMuRF conversion for stream normalization. · `src/smurf/core/conventers/Header2Smurf.cpp`
- [Processor](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/processors/SmurfProcessor.cpp): C++ processing · Native stream processing and analysis pipeline. · `src/smurf/core/processors/SmurfProcessor.cpp`
- [Frame stats](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/counters/FrameStatistics.cpp): C++ counters · Low-level frame statistics and accounting. · `src/smurf/core/counters/FrameStatistics.cpp`
- [Native emulators](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/emulators/StreamDataEmulator.cpp): C++ sources · Synthetic and file-backed stream sources in the native core. · `src/smurf/core/emulators/StreamDataEmulator.cpp`
- [Native tx](https://github.com/slaclab/pysmurf/blob/main/src/smurf/core/transmitters/BaseTransmitter.cpp): C++ transmitters · Native transmitters and channels for downstream delivery. · `src/smurf/core/transmitters/BaseTransmitter.cpp`

### Deployment and support

Configuration, server launch, packaging, docs, and validation.

- [Config files](https://github.com/slaclab/pysmurf/blob/main/cfg_files/template/template.cfg): Site config · Experiment- and site-specific startup and deployment configuration. · `cfg_files/template/template.cfg`
- [Server scripts](https://github.com/slaclab/pysmurf/blob/main/server_scripts/emulate.py): Launch entrypoints · Backend launch scripts for Ethernet, PCIe, and emulation modes. · `server_scripts/emulate.py`
- [Docs](https://github.com/slaclab/pysmurf/blob/main/docs/index.rst): Sphinx docs · Documentation organized around client, core, and user workflows. · `docs/index.rst`
- [Core tests](https://github.com/slaclab/pysmurf/blob/main/tests/core/validate_base_tx.py): Validation · Validation of processor, filter, unwrapper, and transmit behaviors. · `tests/core/validate_base_tx.py`

## Connections

- Client base → Roots: selects backend · Client control objects instantiate the appropriate root for deployment mode.
- Commands → Client base: uses · Command layer builds on the shared client control abstractions.
- Tune workflows → Client base: drives · Tuning workflows execute through the main control API.
- Debug tools → Client base: analyzes · Debug tools operate on data and state exposed by the control layer.
- Client utils → Core utilities: bridges · Client utilities interact with publisher/receiver coordination.
- Roots → Devices: wires up · Backend roots assemble device and transport adapters.
- Roots → Emulators: can use · Roots can route through emulated sources instead of hardware.
- Devices → Converters: feeds · Device ingress passes frames into normalization/conversion.
- Converters → Counters: updates · Normalized frames are counted and summarized.
- Converters → Transmitters: forwards · Converted stream data is forwarded to transmit/sink components.
- Counters → Core utilities: publishes · Stream statistics are distributed through coordination utilities.
- Emulators → Devices: emulates · Synthetic or file-backed sources stand in for live devices.
- Transmitters → Core utilities: delivers · Transmitters connect data sinks to publisher and receiver utilities.
- Core utilities → Debug tools: supports · Utility-layer outputs are consumed by client-side diagnostics.
- Roots → Server scripts: launched by · Server launch scripts start the chosen backend/root variant.
- Config files → Roots: parameterizes · Site and experiment configuration drives backend setup.
- Server scripts → Config files: loads · Launch scripts choose configuration files for a deployment.
- Packet types → Header convert: defines · Shared packet types underpin conversion logic.
- Packet types → Processor: defines · Shared packet types feed native processing.
- Header convert → Processor: prepares · Converted headers and packets are handed to the processor.
- Processor → Frame stats: updates · Processing produces statistics and frame accounting.
- Native emulators → Processor: feeds · Native emulators provide input to the processing pipeline.
- Processor → Native tx: outputs · Processed data can be sent onward through native transmitters.
- Core tests → Processor: validates · Tests exercise core processor behavior.
- Core tests → Native tx: validates · Tests check transmitter behavior and framing.
- Docs → Client base: documents · Documentation mirrors the client API structure.
- Docs → Roots: documents · Documentation covers backend roots and deployment modes.
- Docs → Processor: documents · Documentation explains the native streaming pipeline.
