How arroyosystems/arroyo works
Main Components
• Backend Services:
• arroyo-api: Provides the HTTP/gRPC API endpoints that external clients (including the web UI) use.
• arroyo-controller: Acts as the orchestrator for scheduling, state management, and lifecycle control of the pipelines.
• arroyo-worker & arroyo-node: These modules are the engine’s execution and processing units, carrying out the user-defined computations.
• arroyo-compiler-service & arroyo-planner: Responsible for compiling the SQL/Rust pipeline definitions, planning execution, and transforming queries into execution plans.
• arroyo-rpc: Provides the RPC mechanisms for inter-service communication between components.
• arroyo-server-common: Holds shared logic and utilities for server-side code.
• arroyo-state & arroyo-storage: Manage state handling, checkpointing, fault-recovery, and persisting data (as well as interfacing with external persistent stores).
• Connector and Plugins:
• arroyo-connectors: Provides connectors to various external data sources and sinks (like Kafka, MQTT, filesystem, etc.) to integrate with external systems.
• arroyo-udf (plus its sub-crates, e.g., arroyo-udf-host, arroyo-udf-macros, arroyo-udf-plugin, arroyo-udf-python): These implement extensible user-defined functions (UDFs) that allow users to inject custom code into pipelines.
• Data Processing & Formats:
• arroyo-datastream: Likely handles data stream abstractions.
• arroyo-formats: Manages serialization/deserialization techniques (Avro, JSON, Protocol Buffers) needed by the engine.
• Observability & Supporting Services:
• arroyo-metrics: Collects and reports metrics relevant to pipeline performance.
• arroyo-openapi: Helps with API documentation and possibly client generation.
• arroyo-operator: Encapsulates the stream processing operators (stateful operators like windows, joins, etc.) used within execution flows.
• User Interface:
• webui: The web-based user interface that allows users to interact with the system, create pipelines, and view real-time data results.
• Deployment & DevOps:
• Kubernetes (k8s), Docker-related files in the docker/ directory, and CI configurations (in .github/workflows) support cloud-native deployments, automated testing, and CI/CD pipelines.
Component Relationships and Interactions
• The web UI (frontend) interacts directly with the API service (arroyo-api).
• The API service provides endpoints that communicate with the controller to start, stop, and monitor streaming jobs.
• The controller handles orchestration and delegates processing tasks to worker/node components.
• The planner and compiler service work together to translate and optimize pipeline definitions (in SQL or Rust) into executable plans that the worker nodes execute.
• The connectors serve as bridges between the system and external data sources/sinks. They feed streaming input into the workers and handle output.
• State management (via arroyo-state and arroyo-storage) is integrated across workers to enable checkpointing and recovery in case of failures.
• The RPC layer (arroyo-rpc) facilitates communication among distributed components.
• Observability (using arroyo-metrics) provides monitoring information to both operators and possibly the web UI.
• UDF infrastructure modules allow plugging in custom functions, while the operator crate integrates processing logic (like joins and aggregations).
Architectural Patterns and Design Principles
• Modular, Microservices-style design: Each functionality is split into its dedicated crate, enabling separation of concerns.
• Cloud-native and distributed system design: Built to run in modern containerized environments (Kubernetes manifests present) with scalable worker nodes.
• Dataflow programming model: Emphasizing real-time stream processing, stateful transformations, and incremental computations.
• Extensibility through connectors and UDF mechanisms.
Relevant Technologies, Frameworks, and Libraries
• Rust (Cargo workspaces, multiple crates)
• SQL parsing and planning layers for streaming queries
• Web frontend built with TypeScript/React (under webui)
• Kubernetes and Docker for deployment
1. A Frontend layer (Web UI).
2. A Backend layer divided into:
• API and Gateway (arroyo-api)
• Orchestration and Control (arroyo-controller, planner, compiler service)
• Execution layer (arroyo-worker, arroyo-node, operator)
• Supportive services (state, storage, metrics, openapi)
3. Horizontal integration across the system with Connectors (arroyo-connectors) that interface with external data sources and sinks.
4. Supplementary modules for UDF management and extensions.
• Provide clear directional flow for job requests, stream data ingestion, state updates, and monitoring feedback.