How daq-tools/lorrystream works
1. What this repository is
LorryStream is a Python stream-relay library and CLI. Its core job is to move records between heterogeneous endpoints — for example MQTT, AMQP, Kinesis, and SQL databases — while optionally transforming data in-flight. The project is centered on a generic “source → pipeline → sink” model rather than a single application type.
2. Primary runtime entry points
The user-facing surface is split between a command-line interface and a library API. The README shows two canonical modes: lorry relay <source> <sink> for direct source-to-sink forwarding, and lorry launch for a more pipeline-like syntax. The package also exposes parse_launch(...) for programmatic use, so the same pipeline definition can be driven from code or from the shell.
3. Core orchestration layer
The root package (lorrystream/core.py, model.py, exceptions.py) is the coordination boundary. This layer likely owns pipeline parsing, validation, and runtime setup, turning a URI- or DSL-based description into executable stream components. The CLI modules (cli.py, cmd.py) sit on top of this layer and translate user input into the core model.
4. Stream processing engine
LorryStream is built on top of Streamz and Dask. The lorrystream/streamz package contains the adapter layer that maps LorryStream concepts onto Streamz primitives. This is the in-process dataflow engine boundary: it is where records are pulled from sources, optionally transformed, and pushed into sinks.
5. Source-side adapters
Source implementations are separated from the orchestration layer. The streamz/sources.py and protocol-specific modules such as streamz/amqp.py and the docs under doc/source/ indicate support for message buses and network listeners. The README explicitly names AMQP, Kafka, MQTT, ZeroMQ, TCP, UDP, HTTP, and WebSocket as supported source categories, so the architecture includes multiple ingress adapters behind a uniform abstraction.
6. Sink-side adapters
Sink implementations are similarly isolated in streamz/sinks.py and related model code. The README emphasizes two sink families: SQL databases via SQLAlchemy and other message brokers. Examples show CrateDB as a concrete SQL sink target, so a major boundary in the architecture is “stream record → database row” versus “stream record → outbound message”.
7. Transformation / pipeline composition
The repository is not just a transport relay; it also supports intermediate processing steps. The examples (examples/transform.py, examples/parse_launch.py, examples/appsink.py) and the pipeline-style launch syntax suggest that components can be composed into multi-stage flows, with the engine acting as a graph of connected stream elements rather than a single point-to-point connector.
8. AWS integration subsystem: Carabas
The lorrystream/carabas subtree is a distinct infrastructure/deployment subsystem for AWS. It contains CloudFormation-related code (aws/cf), stack definitions (aws/stack/dms.py, aws/stack/dynamodb.py), and Lambda/function packaging (aws/function/*). This part of the repo is about provisioning and packaging stream backplanes on AWS, not about the in-process stream engine itself.
9. AWS dataflow patterns supported
The AWS examples and docs indicate concrete deployment patterns involving DynamoDB, Kinesis, Lambda, and DMS. The names dynamodb_kinesis_lambda_oci_cratedb.py and rds_postgresql_kinesis_lambda_oci_cratedb.py show end-to-end flows where AWS data sources are captured, processed via Lambda, and delivered to CrateDB, with OCI packaging used to distribute the Lambda/runtime artifact.
10. Experimental and reference implementations
The spike/ tree contains standalone prototypes for KCL-based Kinesis and DynamoDB consumers. These are clearly separated from the main library, which suggests they exist as reference code, experiments, or proof-of-concept implementations rather than as the primary runtime path.
11. Utilities and packaging support
The util/ package provides supporting functionality such as CLI helpers, async helpers, data helpers, and Python bundling/PEP 723 support. These are cross-cutting utilities used by the higher-level CLI, packaging, and deployment code rather than by a single stream adapter.
12. Examples, tests, and docs as architecture guides
The examples/ tree demonstrates the intended deployment shapes, while tests/ includes fixtures for AMQP and LocalStack plus fixture data for Kinesis/DynamoDB-like payloads. The documentation under doc/ is unusually rich and maps directly onto the supported source/sink families, so it effectively describes the intended subsystem boundaries and integration points.
13. Big-picture architecture boundary
At a high level, the repo separates into three layers: a user-facing command/API layer, a Streamz-based runtime pipeline layer, and an AWS deployment/packaging layer. External systems — MQTT/AMQP/Kinesis/DynamoDB/DMS/SQL databases/CrateDB — connect only through adapter boundaries, which is the main architectural principle to reflect in a diagram.
Main entry points: core.py Core · streamz/__init__.py Streamz · carabas/__init__.py Carabas · util/__init__.py Utilities
Components 18
User-facing
Command-line and programmatic entry points
Stream runtime
Core pipeline model and Streamz-based execution
- Core · Pipeline orchestration and runtime setup
lorrystream/core.py - Model · Shared configuration and launch model definitions
lorrystream/model.py - Exceptions · Repository-specific exception types
lorrystream/exceptions.py - Streamz · Bridge between LorryStream concepts and Streamz primitives
lorrystream/streamz/__init__.py - Stream model · Execution model used by the adapter layer
lorrystream/streamz/model.py - Sources · Uniform source abstractions for message and network inputs
lorrystream/streamz/sources.py - Sinks · Uniform sink abstractions for databases and outbound brokers
lorrystream/streamz/sinks.py - AMQP · AMQP-specific source integration
lorrystream/streamz/amqp.py
AWS Carabas
AWS deployment, packaging, and dataflow infrastructure
- Carabas · AWS deployment and packaging subsystem
lorrystream/carabas/__init__.py - CloudFormation · CloudFormation-related helpers and stack composition
lorrystream/carabas/aws/cf/__init__.py - Stacks · Reusable AWS stack definitions for DMS and DynamoDB
lorrystream/carabas/aws/stack/__init__.py - Lambda pkg · Lambda function model and OCI/zip packaging helpers
lorrystream/carabas/aws/function/__init__.py
Supporting code
Utilities, examples, and experiments
- Utilities · Cross-cutting CLI, async, data, and packaging helpers
lorrystream/util/__init__.py - Processes · Named end-to-end stream processing compositions
lorrystream/process/__init__.py - Examples · Runnable examples for transforms and AWS flows
examples/transform.py - Spikes · Standalone KCL-based proof-of-concept implementations
lorrystream/spike/__init__.py
Connections 19
- CLI → Commands: dispatches · CLI routes subcommands into command handling
- Commands → Core: invokes · Command parsing hands off to core orchestration
- Core → Model: uses · Core builds and validates pipeline models
- Core → Exceptions: raises · Runtime and validation errors flow through repository exceptions
- Core → Streamz: builds · Core turns launch descriptions into Streamz execution graphs
- Streamz → Stream model: adapts · Adapter layer maps repository model to Streamz runtime model
- Streamz → Sources: connects · Streamz layer constructs ingress endpoints
- Streamz → Sinks: connects · Streamz layer constructs egress endpoints
- Sources → AMQP: specializes · AMQP is one concrete ingress adapter
- Streamz → Utilities: uses · Runtime and adapter code rely on shared helpers
- CLI → Utilities: uses · CLI support code is centralized in utilities
- Processes → Core: composes · Named process recipes are assembled on top of core orchestration
- Examples → Core: demonstrates · Examples show direct and launched pipeline usage
- Examples → Carabas: demonstrates · AWS examples exercise the deployment subsystem
- Carabas → CloudFormation: contains · Carabas includes CloudFormation-oriented infrastructure code
- Carabas → Stacks: contains · Carabas provides reusable AWS stack definitions
- Carabas → Lambda pkg: contains · Carabas packages Lambda artifacts and runtimes
- Carabas → Utilities: uses · Deployment code reuses common helpers and packaging utilities
- Spikes → Utilities: uses · Experimental prototypes depend on shared utilities and support code