# daq-tools/lorrystream architecture

> Architecture diagram of the GitHub repository daq-tools/lorrystream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daq-tools/lorrystream
- Repository: https://github.com/daq-tools/lorrystream

## Overview

1. **What this repository is**
   LorryStream is a Python stream-relay library and CLI. Its core job is to move records between heterogeneous endpoints — for example MQTT, AMQP, Kinesis, and SQL databases — while optionally transforming data in-flight. The project is centered on a generic “source → pipeline → sink” model rather than a single application type.

2. **Primary runtime entry points**
   The user-facing surface is split between a command-line interface and a library API. The README shows two canonical modes: `lorry relay <source> <sink>` for direct source-to-sink forwarding, and `lorry launch` for a more pipeline-like syntax. The package also exposes `parse_launch(...)` for programmatic use, so the same pipeline definition can be driven from code or from the shell.

3. **Core orchestration layer**
   The root package (`lorrystream/core.py`, `model.py`, `exceptions.py`) is the coordination boundary. This layer likely owns pipeline parsing, validation, and runtime setup, turning a URI- or DSL-based description into executable stream components. The CLI modules (`cli.py`, `cmd.py`) sit on top of this layer and translate user input into the core model.

4. **Stream processing engine**
   LorryStream is built on top of **Streamz** and **Dask**. The `lorrystream/streamz` package contains the adapter layer that maps LorryStream concepts onto Streamz primitives. This is the in-process dataflow engine boundary: it is where records are pulled from sources, optionally transformed, and pushed into sinks.

5. **Source-side adapters**
   Source implementations are separated from the orchestration layer. The `streamz/sources.py` and protocol-specific modules such as `streamz/amqp.py` and the docs under `doc/source/` indicate support for message buses and network listeners. The README explicitly names AMQP, Kafka, MQTT, ZeroMQ, TCP, UDP, HTTP, and WebSocket as supported source categories, so the architecture includes multiple ingress adapters behind a uniform abstraction.

6. **Sink-side adapters**
   Sink implementations are similarly isolated in `streamz/sinks.py` and related model code. The README emphasizes two sink families: SQL databases via **SQLAlchemy** and other message brokers. Examples show CrateDB as a concrete SQL sink target, so a major boundary in the architecture is “stream record → database row” versus “stream record → outbound message”.

7. **Transformation / pipeline composition**
   The repository is not just a transport relay; it also supports intermediate processing steps. The examples (`examples/transform.py`, `examples/parse_launch.py`, `examples/appsink.py`) and the pipeline-style `launch` syntax suggest that components can be composed into multi-stage flows, with the engine acting as a graph of connected stream elements rather than a single point-to-point connector.

8. **AWS integration subsystem: Carabas**
   The `lorrystream/carabas` subtree is a distinct infrastructure/deployment subsystem for AWS. It contains CloudFormation-related code (`aws/cf`), stack definitions (`aws/stack/dms.py`, `aws/stack/dynamodb.py`), and Lambda/function packaging (`aws/function/*`). This part of the repo is about provisioning and packaging stream backplanes on AWS, not about the in-process stream engine itself.

9. **AWS dataflow patterns supported**
   The AWS examples and docs indicate concrete deployment patterns involving **DynamoDB**, **Kinesis**, **Lambda**, and **DMS**. The names `dynamodb_kinesis_lambda_oci_cratedb.py` and `rds_postgresql_kinesis_lambda_oci_cratedb.py` show end-to-end flows where AWS data sources are captured, processed via Lambda, and delivered to CrateDB, with OCI packaging used to distribute the Lambda/runtime artifact.

10. **Experimental and reference implementations**
   The `spike/` tree contains standalone prototypes for KCL-based Kinesis and DynamoDB consumers. These are clearly separated from the main library, which suggests they exist as reference code, experiments, or proof-of-concept implementations rather than as the primary runtime path.

11. **Utilities and packaging support**
   The `util/` package provides supporting functionality such as CLI helpers, async helpers, data helpers, and Python bundling/PEP 723 support. These are cross-cutting utilities used by the higher-level CLI, packaging, and deployment code rather than by a single stream adapter.

12. **Examples, tests, and docs as architecture guides**
   The `examples/` tree demonstrates the intended deployment shapes, while `tests/` includes fixtures for AMQP and LocalStack plus fixture data for Kinesis/DynamoDB-like payloads. The documentation under `doc/` is unusually rich and maps directly onto the supported source/sink families, so it effectively describes the intended subsystem boundaries and integration points.

13. **Big-picture architecture boundary**
   At a high level, the repo separates into three layers: a user-facing command/API layer, a Streamz-based runtime pipeline layer, and an AWS deployment/packaging layer. External systems — MQTT/AMQP/Kinesis/DynamoDB/DMS/SQL databases/CrateDB — connect only through adapter boundaries, which is the main architectural principle to reflect in a diagram.

Main entry points: [`lorrystream/core.py`](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/core.py) (Core), [`lorrystream/streamz/__init__.py`](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/__init__.py) (Streamz), [`lorrystream/carabas/__init__.py`](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/__init__.py) (Carabas), [`lorrystream/util/__init__.py`](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/util/__init__.py) (Utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_cli["User-facing"]
  node_n_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_n_cmd["Commands<br/>arg parsing<br/>[cmd.py]"]
end

subgraph group_group_runtime["Stream runtime"]
  node_n_core["Core<br/>orchestrator<br/>[core.py]"]
  node_n_model["Model<br/>[model.py]"]
  node_n_exc["Exceptions<br/>errors<br/>[exceptions.py]"]
  node_n_streamz_pkg["Streamz<br/>adapter layer<br/>[__init__.py]"]
  node_n_streamz_model["Stream model<br/>[model.py]"]
  node_n_sources["Sources<br/>ingress adapters<br/>[sources.py]"]
  node_n_sinks["Sinks<br/>egress adapters<br/>[sinks.py]"]
  node_n_amqp["AMQP<br/>protocol source<br/>[amqp.py]"]
end

subgraph group_group_aws["AWS Carabas"]
  node_n_carabas["Carabas<br/>aws infra<br/>[__init__.py]"]
  node_n_cf["CloudFormation<br/>infra templates<br/>[__init__.py]"]
  node_n_stack["Stacks<br/>[__init__.py]"]
  node_n_function["Lambda pkg<br/>packaging<br/>[__init__.py]"]
end

subgraph group_group_support["Supporting code"]
  node_n_util["Utilities<br/>helpers<br/>[__init__.py]"]
  node_n_process["Processes<br/>pipeline recipes<br/>[__init__.py]"]
  node_n_examples["Examples<br/>usage samples<br/>[transform.py]"]
  node_n_spike["Spikes<br/>prototypes<br/>[__init__.py]"]
end

node_n_cli -->|"dispatches"| node_n_cmd
node_n_cmd -->|"invokes"| node_n_core
node_n_core -->|"uses"| node_n_model
node_n_core -->|"raises"| node_n_exc
node_n_core -->|"builds"| node_n_streamz_pkg
node_n_streamz_pkg -->|"adapts"| node_n_streamz_model
node_n_streamz_pkg -->|"connects"| node_n_sources
node_n_streamz_pkg -->|"connects"| node_n_sinks
node_n_sources -->|"specializes"| node_n_amqp
node_n_streamz_pkg -->|"uses"| node_n_util
node_n_cli -->|"uses"| node_n_util
node_n_process -->|"composes"| node_n_core
node_n_examples -->|"demonstrates"| node_n_core
node_n_examples -->|"demonstrates"| node_n_carabas
node_n_carabas -->|"contains"| node_n_cf
node_n_carabas -->|"contains"| node_n_stack
node_n_carabas -->|"contains"| node_n_function
node_n_carabas -->|"uses"| node_n_util
node_n_spike -->|"uses"| node_n_util

click node_n_cli "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/cli.py"
click node_n_cmd "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/cmd.py"
click node_n_core "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/core.py"
click node_n_model "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/model.py"
click node_n_exc "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/exceptions.py"
click node_n_streamz_pkg "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/__init__.py"
click node_n_streamz_model "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/model.py"
click node_n_sources "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/sources.py"
click node_n_sinks "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/sinks.py"
click node_n_amqp "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/amqp.py"
click node_n_util "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/util/__init__.py"
click node_n_process "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/process/__init__.py"
click node_n_examples "https://github.com/daq-tools/lorrystream/blob/main/examples/transform.py"
click node_n_carabas "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/__init__.py"
click node_n_cf "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/cf/__init__.py"
click node_n_stack "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/stack/__init__.py"
click node_n_function "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/function/__init__.py"
click node_n_spike "https://github.com/daq-tools/lorrystream/blob/main/lorrystream/spike/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_cli,node_n_cmd toneBlue
class node_n_core,node_n_model,node_n_exc,node_n_streamz_pkg,node_n_streamz_model,node_n_sources,node_n_sinks,node_n_amqp toneAmber
class node_n_carabas,node_n_cf,node_n_stack,node_n_function toneMint
class node_n_util,node_n_process,node_n_examples,node_n_spike toneRose
```

## Components

### User-facing

Command-line and programmatic entry points

- [CLI](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/cli.py): entrypoint · Command-line interface for relay and launch modes · `lorrystream/cli.py`
- [Commands](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/cmd.py): arg parsing · Subcommand and command dispatch layer · `lorrystream/cmd.py`

### Stream runtime

Core pipeline model and Streamz-based execution

- [Core](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/core.py): orchestrator · Pipeline orchestration and runtime setup · `lorrystream/core.py`
- [Model](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/model.py): pipeline model · Shared configuration and launch model definitions · `lorrystream/model.py`
- [Exceptions](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/exceptions.py): errors · Repository-specific exception types · `lorrystream/exceptions.py`
- [Streamz](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/__init__.py): adapter layer · Bridge between LorryStream concepts and Streamz primitives · `lorrystream/streamz/__init__.py`
- [Stream model](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/model.py): stream model · Execution model used by the adapter layer · `lorrystream/streamz/model.py`
- [Sources](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/sources.py): ingress adapters · Uniform source abstractions for message and network inputs · `lorrystream/streamz/sources.py`
- [Sinks](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/sinks.py): egress adapters · Uniform sink abstractions for databases and outbound brokers · `lorrystream/streamz/sinks.py`
- [AMQP](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/streamz/amqp.py): protocol source · AMQP-specific source integration · `lorrystream/streamz/amqp.py`

### AWS Carabas

AWS deployment, packaging, and dataflow infrastructure

- [Carabas](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/__init__.py): aws infra · AWS deployment and packaging subsystem · `lorrystream/carabas/__init__.py`
- [CloudFormation](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/cf/__init__.py): infra templates · CloudFormation-related helpers and stack composition · `lorrystream/carabas/aws/cf/__init__.py`
- [Stacks](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/stack/__init__.py): aws stacks · Reusable AWS stack definitions for DMS and DynamoDB · `lorrystream/carabas/aws/stack/__init__.py`
- [Lambda pkg](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/carabas/aws/function/__init__.py): packaging · Lambda function model and OCI/zip packaging helpers · `lorrystream/carabas/aws/function/__init__.py`

### Supporting code

Utilities, examples, and experiments

- [Utilities](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/util/__init__.py): helpers · Cross-cutting CLI, async, data, and packaging helpers · `lorrystream/util/__init__.py`
- [Processes](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/process/__init__.py): pipeline recipes · Named end-to-end stream processing compositions · `lorrystream/process/__init__.py`
- [Examples](https://github.com/daq-tools/lorrystream/blob/main/examples/transform.py): usage samples · Runnable examples for transforms and AWS flows · `examples/transform.py`
- [Spikes](https://github.com/daq-tools/lorrystream/blob/main/lorrystream/spike/__init__.py): prototypes · Standalone KCL-based proof-of-concept implementations · `lorrystream/spike/__init__.py`

## Connections

- CLI → Commands: dispatches · CLI routes subcommands into command handling
- Commands → Core: invokes · Command parsing hands off to core orchestration
- Core → Model: uses · Core builds and validates pipeline models
- Core → Exceptions: raises · Runtime and validation errors flow through repository exceptions
- Core → Streamz: builds · Core turns launch descriptions into Streamz execution graphs
- Streamz → Stream model: adapts · Adapter layer maps repository model to Streamz runtime model
- Streamz → Sources: connects · Streamz layer constructs ingress endpoints
- Streamz → Sinks: connects · Streamz layer constructs egress endpoints
- Sources → AMQP: specializes · AMQP is one concrete ingress adapter
- Streamz → Utilities: uses · Runtime and adapter code rely on shared helpers
- CLI → Utilities: uses · CLI support code is centralized in utilities
- Processes → Core: composes · Named process recipes are assembled on top of core orchestration
- Examples → Core: demonstrates · Examples show direct and launched pipeline usage
- Examples → Carabas: demonstrates · AWS examples exercise the deployment subsystem
- Carabas → CloudFormation: contains · Carabas includes CloudFormation-oriented infrastructure code
- Carabas → Stacks: contains · Carabas provides reusable AWS stack definitions
- Carabas → Lambda pkg: contains · Carabas packages Lambda artifacts and runtimes
- Carabas → Utilities: uses · Deployment code reuses common helpers and packaging utilities
- Spikes → Utilities: uses · Experimental prototypes depend on shared utilities and support code
