How slaclab/pysmurf works
1. Repository purpose
- This is the pysmurf control stack for SMuRF hardware: a Python-centric system for operating readout hardware, configuring experiments, streaming data, and running analysis/tuning workflows.
- It is not a single app; it is a layered toolkit spanning control, hardware I/O, data processing, emulation, and operational scripts.
2. Main runtime split: Python control layer + C++ core
- The user-facing package lives under python/pysmurf, while performance-critical streaming/packet logic is implemented in C++ under src/smurf with headers in include/smurf.
- The Python package exposes higher-level control and analysis APIs; the C++ library handles low-level data handling, packet conversion, counters, and transmitters.
3. Primary user-facing subsystem
- python/pysmurf/client is the operator-facing API and command layer.
- It contains:
- base/ for the main control abstractions, config handling, and logging.
- command/ for executable-style hardware commands and operational utilities.
- tune/ for tuning workflows.
- debug/ for diagnostics and experiment-specific analysis helpers.
- util/ and watchdog/ for supporting tools and health checks.
- This is the layer most likely imported by experiment scripts and run interactively.
4. Hardware/backend abstraction
- python/pysmurf/core/roots defines the backend entry points for different deployment modes:
- Ethernet vs PCIe variants.
- Dev board vs cryogenic module variants.
- EmulationRoot for simulated operation.
- This is the architectural boundary between the Python control world and the actual SMuRF hardware transport.
5. Data acquisition and stream-processing pipeline
- The core streaming path is organized as:
- device input (PcieCard, UdpReceiver, SmurfApplication)
- packet/header normalization (Header2Smurf, SmurfHeader, SmurfPacket)
- processing/statistics (SmurfProcessor, FrameStatistics)
- output/transmission (BaseTransmitter, DataToFile, transmitter channels)
- The design suggests a pipeline architecture where incoming frames are converted into a SMuRF-specific representation, analyzed, and then forwarded to sinks such as files or downstream consumers.
6. Emulation and offline data support
- python/pysmurf/core/emulators and src/smurf/core/emulators provide simulated stream sources, including file-backed and synthetic data generation.
- This allows the same processing stack to run without hardware, which is important for testing, development, and demos.
- The presence of StreamDataSource, StreamDataEmulatorI16/I32, and DataFromFile indicates an explicit offline/virtual data path parallel to the live hardware path.
7. Publisher/receiver and utility layer
- python/pysmurf/core/utilities includes support components like a data receiver, setup grouping, and a publisher.
- These pieces look like the integration boundary for distributing processed stream data and coordinating system initialization.
- They sit between the low-level core pipeline and higher-level client workflows.
8. Server-side launch scripts
- server_scripts/ and python/pysmurf/core/server_scripts/ provide entry points for starting different backend configurations.
- The top-level scripts map to the root/backend variants in the codebase, such as Ethernet/PCIe and emulation modes.
- This indicates the server process is launched separately from the client and selected by deployment environment.
9. Configuration is experiment/site driven
- cfg_files/ contains many site- and experiment-specific .cfg files.
- These are not generic defaults; they encode deployment-specific SMuRF startup and experiment settings for different labs and instruments.
- In architecture terms, config files are a major control-plane input that parameterize the same software stack for different hardware installations.
10. Deployment and packaging
- The repo is packaged as a Python distribution (pyproject.toml, requirements.txt) with a compiled C++ extension/build system (CMakeLists.txt, src/, include/).
- Docker support is split into client and server images under docker/client and docker/server, reflecting the separation between operator tooling and hardware-facing services.
- smurfConfig.cmake.in and releaseGen.py suggest release packaging and native-library build integration.
11. Documentation and validation
- docs/ contains Sphinx documentation divided into client, core, and user sections, mirroring the code architecture.
- tests/core/ validates key low-level processing behavior such as the processor, filter, and unwrapper logic.
- This reinforces that the C++ core is the most algorithmically sensitive part of the system.
12. Operational and exploratory code
- scratch/ contains many one-off analysis scripts, notebooks, and experiment prototypes.
- These are useful for users but are not core architecture; they sit outside the supported product path and should be treated as experimental tooling.
13. Overall data/control flow
- Typical flow is: operator script or client API → configured root/backend → hardware or emulator input stream → C++ processing/conversion pipeline → transmitter/publisher/file output → analysis/debug/tuning tooling.
- The main architectural boundary is between the Python control plane and the native streaming pipeline, with hardware/emulation as interchangeable sources behind the same root abstractions.
14. What to show in an architecture diagram
- Show distinct boxes for: client/control API, backend root/transport selection, data acquisition/processing core, output/transmit/publish sinks, configuration inputs, and emulation/offline sources.
- Also show deployment separation between client and server containers/processes, and the Python/C++ implementation boundary inside the core stack.
Main entry points: SmurfProcessor.cpp Processor · Common.py Roots · base_class.py Client base · _SmurfPublisher.py Core utilities
Components 22
Python control plane
Operator-facing APIs, commands, tuning, diagnostics, and utilities.
- Client base · Main control abstractions, config, logging, and control state.
python/pysmurf/client/base/base_class.py - Commands · Executable-style hardware commands and operational utilities.
python/pysmurf/client/command/smurf_command.py - Tune workflows · Tuning and calibration workflows used during operation.
python/pysmurf/client/tune/smurf_tune.py - Debug tools · Experiment-specific analysis and diagnostic helpers.
python/pysmurf/client/debug/smurf_noise.py - Client utils · File reading, listening, publishing, and small operational helpers.
python/pysmurf/client/util/pub.py
Streaming core
Backend roots, device adapters, emulation, processing, and sinks.
- Roots · Entry points for Ethernet, PCIe, dev-board, cryogenic, and emulated deployments.
python/pysmurf/core/roots/Common.py - Devices · Python-facing adapters for card, UDP receiver, application, and processor objects.
python/pysmurf/core/devices/_SmurfApplication.py - Converters · Transforms device headers and frames into SMuRF-specific stream objects.
python/pysmurf/core/conventers/_Header2Smurf.py - Counters · Frame statistics and stream counters used for monitoring and validation.
python/pysmurf/core/counters/_FrameStatistics.py - Emulators · Synthetic and file-backed data sources for simulated operation.
python/pysmurf/core/emulators/_StreamDataSource.py - Transmitters · Python-side transmitters including file output and publisher integration.
python/pysmurf/core/transmitters/_DataToFile.py - Core utilities · Setup groups, data receiver, and publisher coordination utilities.
python/pysmurf/core/utilities/_SmurfPublisher.py
Native C++ core
Performance-critical packet, processing, and transmitter implementation.
- Packet types · Smurf headers, packets, timing, and shared native data structures.
include/smurf/core/common/SmurfPacket.h - Header convert · Native header-to-SMuRF conversion for stream normalization.
src/smurf/core/conventers/Header2Smurf.cpp - Processor · Native stream processing and analysis pipeline.
src/smurf/core/processors/SmurfProcessor.cpp - Frame stats · Low-level frame statistics and accounting.
src/smurf/core/counters/FrameStatistics.cpp - Native emulators · Synthetic and file-backed stream sources in the native core.
src/smurf/core/emulators/StreamDataEmulator.cpp - Native tx · Native transmitters and channels for downstream delivery.
src/smurf/core/transmitters/BaseTransmitter.cpp
Deployment and support
Configuration, server launch, packaging, docs, and validation.
- Config files · Experiment- and site-specific startup and deployment configuration.
cfg_files/template/template.cfg - Server scripts · Backend launch scripts for Ethernet, PCIe, and emulation modes.
server_scripts/emulate.py - Docs · Documentation organized around client, core, and user workflows.
docs/index.rst - Core tests · Validation of processor, filter, unwrapper, and transmit behaviors.
tests/core/validate_base_tx.py
Connections 28
- Client base → Roots: selects backend · Client control objects instantiate the appropriate root for deployment mode.
- Commands → Client base: uses · Command layer builds on the shared client control abstractions.
- Tune workflows → Client base: drives · Tuning workflows execute through the main control API.
- Debug tools → Client base: analyzes · Debug tools operate on data and state exposed by the control layer.
- Client utils → Core utilities: bridges · Client utilities interact with publisher/receiver coordination.
- Roots → Devices: wires up · Backend roots assemble device and transport adapters.
- Roots → Emulators: can use · Roots can route through emulated sources instead of hardware.
- Devices → Converters: feeds · Device ingress passes frames into normalization/conversion.
- Converters → Counters: updates · Normalized frames are counted and summarized.
- Converters → Transmitters: forwards · Converted stream data is forwarded to transmit/sink components.
- Counters → Core utilities: publishes · Stream statistics are distributed through coordination utilities.
- Emulators → Devices: emulates · Synthetic or file-backed sources stand in for live devices.
- Transmitters → Core utilities: delivers · Transmitters connect data sinks to publisher and receiver utilities.
- Core utilities → Debug tools: supports · Utility-layer outputs are consumed by client-side diagnostics.
- Roots → Server scripts: launched by · Server launch scripts start the chosen backend/root variant.
- Config files → Roots: parameterizes · Site and experiment configuration drives backend setup.
- Server scripts → Config files: loads · Launch scripts choose configuration files for a deployment.
- Packet types → Header convert: defines · Shared packet types underpin conversion logic.
- Packet types → Processor: defines · Shared packet types feed native processing.
- Header convert → Processor: prepares · Converted headers and packets are handed to the processor.
- Processor → Frame stats: updates · Processing produces statistics and frame accounting.
- Native emulators → Processor: feeds · Native emulators provide input to the processing pipeline.
- Processor → Native tx: outputs · Processed data can be sent onward through native transmitters.
- Core tests → Processor: validates · Tests exercise core processor behavior.
- Core tests → Native tx: validates · Tests check transmitter behavior and framing.
- Docs → Client base: documents · Documentation mirrors the client API structure.
- Docs → Roots: documents · Documentation covers backend roots and deployment modes.
- Docs → Processor: documents · Documentation explains the native streaming pipeline.