# arroyosystems/arroyo architecture

> Architecture diagram of the GitHub repository arroyosystems/arroyo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arroyosystems/arroyo
- Repository: https://github.com/arroyosystems/arroyo

## Overview

##### Main Components
- Backend Services:
- arroyo-api: Provides the HTTP/gRPC API endpoints that external clients (including the web UI) use.
- arroyo-controller: Acts as the orchestrator for scheduling, state management, and lifecycle control of the pipelines.
- arroyo-worker & arroyo-node: These modules are the engine’s execution and processing units, carrying out the user-defined computations.
- arroyo-compiler-service & arroyo-planner: Responsible for compiling the SQL/Rust pipeline definitions, planning execution, and transforming queries into execution plans.
- arroyo-rpc: Provides the RPC mechanisms for inter-service communication between components.
- arroyo-server-common: Holds shared logic and utilities for server-side code.
- arroyo-state & arroyo-storage: Manage state handling, checkpointing, fault-recovery, and persisting data (as well as interfacing with external persistent stores).

- Connector and Plugins:
- arroyo-connectors: Provides connectors to various external data sources and sinks (like Kafka, MQTT, filesystem, etc.) to integrate with external systems.
- arroyo-udf (plus its sub-crates, e.g., arroyo-udf-host, arroyo-udf-macros, arroyo-udf-plugin, arroyo-udf-python): These implement extensible user-defined functions (UDFs) that allow users to inject custom code into pipelines.

- Data Processing & Formats:
- arroyo-datastream: Likely handles data stream abstractions.
- arroyo-formats: Manages serialization/deserialization techniques (Avro, JSON, Protocol Buffers) needed by the engine.

- Observability & Supporting Services:
- arroyo-metrics: Collects and reports metrics relevant to pipeline performance.
- arroyo-openapi: Helps with API documentation and possibly client generation.
- arroyo-operator: Encapsulates the stream processing operators (stateful operators like windows, joins, etc.) used within execution flows.

- User Interface:
- webui: The web-based user interface that allows users to interact with the system, create pipelines, and view real-time data results.

- Deployment & DevOps:
- Kubernetes (k8s), Docker-related files in the docker/ directory, and CI configurations (in .github/workflows) support cloud-native deployments, automated testing, and CI/CD pipelines.

##### Component Relationships and Interactions
- The web UI (frontend) interacts directly with the API service (arroyo-api).
- The API service provides endpoints that communicate with the controller to start, stop, and monitor streaming jobs.
- The controller handles orchestration and delegates processing tasks to worker/node components.
- The planner and compiler service work together to translate and optimize pipeline definitions (in SQL or Rust) into executable plans that the worker nodes execute.
- The connectors serve as bridges between the system and external data sources/sinks. They feed streaming input into the workers and handle output.
- State management (via arroyo-state and arroyo-storage) is integrated across workers to enable checkpointing and recovery in case of failures.
- The RPC layer (arroyo-rpc) facilitates communication among distributed components.
- Observability (using arroyo-metrics) provides monitoring information to both operators and possibly the web UI.
- UDF infrastructure modules allow plugging in custom functions, while the operator crate integrates processing logic (like joins and aggregations).

##### Architectural Patterns and Design Principles
- Modular, Microservices-style design: Each functionality is split into its dedicated crate, enabling separation of concerns.
- Cloud-native and distributed system design: Built to run in modern containerized environments (Kubernetes manifests present) with scalable worker nodes.
- Dataflow programming model: Emphasizing real-time stream processing, stateful transformations, and incremental computations.
- Extensibility through connectors and UDF mechanisms.

##### Relevant Technologies, Frameworks, and Libraries
- Rust (Cargo workspaces, multiple crates)
- SQL parsing and planning layers for streaming queries
- Web frontend built with TypeScript/React (under webui)
- Kubernetes and Docker for deployment

1. A Frontend layer (Web UI).
2. A Backend layer divided into:
- API and Gateway (arroyo-api)
- Orchestration and Control (arroyo-controller, planner, compiler service)
- Execution layer (arroyo-worker, arroyo-node, operator)
- Supportive services (state, storage, metrics, openapi)
3. Horizontal integration across the system with Connectors (arroyo-connectors) that interface with external data sources and sinks.
4. Supplementary modules for UDF management and extensions.
- Provide clear directional flow for job requests, stream data ingestion, state updates, and monitoring feedback.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Frontend Layer
    subgraph "Frontend"
        UI["Web UI"] 
    end

    %% API Gateway
    subgraph "API & Gateway"
        API["arroyo-api"]
    end

    %% Control & Orchestration Layer
    subgraph "Control & Orchestration"
        CTRL["arroyo-controller"]
        PLNR["arroyo-planner"]
        COMP["arroyo-compiler-service"]
    end

    %% Execution Layer
    subgraph "Execution Layer"
        WORKER["arroyo-worker"]
        OPER["arroyo-operator"]
    end

    %% Data Processing Support
    subgraph "Data Processing Support"
        DATA["arroyo-datastream"]
        FORMATS["arroyo-formats"]
    end

    %% State & Observability
    subgraph "State & Observability"
        STATE["arroyo-state"]
        STORAGE["arroyo-storage"]
        METRICS["arroyo-metrics"]
        OPENAPI["arroyo-openapi"]
    end

    %% Integration & Shared Utilities
    subgraph "Integration & Shared Utilities"
        RPC["arroyo-rpc"]
        COMMON["arroyo-server-common"]
        CONN["arroyo-connectors"]
    end

    %% UDF Framework
    subgraph "UDF Framework"
        UDF_COM["arroyo-udf-common"]
        UDF_HOST["arroyo-udf-host"]
        UDF_MACRO["arroyo-udf-macros"]
        UDF_PLUGIN["arroyo-udf-plugin"]
        UDF_PY["arroyo-udf-python"]
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        DOCKER["Docker & CI/CD"]
    end

    %% Connections
    UI -->|"calls"| API
    API -->|"forwards requests"| CTRL
    API -->|"exposes docs"| OPENAPI
    CTRL -->|"orchestrates"| WORKER
    CTRL -->|"sends planning info"| PLNR
    CTRL -->|"sends compile request"| COMP
    PLNR <-->|"collaborates"| COMP
    WORKER -->|"executes operations"| OPER
    CONN -->|"feeds data"| WORKER
    WORKER -->|"processes stream"| DATA
    WORKER -->|"handles formats"| FORMATS
    WORKER -->|"manages state"| STATE
    WORKER -->|"persists checkpoints"| STORAGE
    WORKER -->|"reports metrics"| METRICS
    WORKER -->|"communicates via"| RPC
    CTRL -->|"communicates via"| RPC
    API -->|"communicates via"| RPC
    CTRL -->|"uses utility"| COMMON
    RPC -->|"supports"| COMMON
    WORKER -->|"utilizes UDFs"| UDF_COM
    UDF_COM --> UDF_HOST
    UDF_COM --> UDF_MACRO
    UDF_COM --> UDF_PLUGIN
    UDF_COM --> UDF_PY
    DOCKER ---|"containerizes"| API
    DOCKER ---|"containerizes"| CTRL
    DOCKER ---|"containerizes"| WORKER

    %% Click Events
    click UI "https://github.com/arroyosystems/arroyo/tree/master/webui"
    click API "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-api"
    click CTRL "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-controller"
    click PLNR "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-planner"
    click COMP "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-compiler-service"
    click WORKER "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-worker"
    click OPER "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-operator"
    click DATA "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-datastream"
    click FORMATS "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-formats"
    click STATE "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-state"
    click STORAGE "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-storage"
    click METRICS "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-metrics"
    click OPENAPI "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-openapi"
    click RPC "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-rpc"
    click COMMON "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-server-common"
    click CONN "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-connectors"
    click UDF_COM "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-common"
    click UDF_HOST "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-host"
    click UDF_MACRO "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-macros"
    click UDF_PLUGIN "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-plugin"
    click UDF_PY "https://github.com/arroyosystems/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-python"
    click DOCKER "https://github.com/arroyosystems/arroyo/tree/master/docker/"

    %% Styles
    classDef frontend fill:#ffedcc,stroke:#cc9900,stroke-width:2px;
    classDef api fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef orchestration fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef execution fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef support fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef integration fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef udf fill:#fce5cd,stroke:#b36b00,stroke-width:2px;
    classDef deployment fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;

    class UI frontend;
    class API api;
    class CTRL,PLNR,COMP orchestration;
    class WORKER,OPER execution;
    class DATA,FORMATS,STATE,STORAGE,METRICS,OPENAPI support;
    class RPC,COMMON,CONN integration;
    class UDF_COM,UDF_HOST,UDF_MACRO,UDF_PLUGIN,UDF_PY udf;
    class DOCKER deployment;
```
