# sudhishp/arroyo architecture

> Architecture diagram of the GitHub repository sudhishp/arroyo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sudhishp/arroyo
- Repository: https://github.com/sudhishp/arroyo

## Overview

##### Project Type and Purpose
- This is a distributed, real-time stream processing engine (Arroyo) written in Rust.
- It provides SQL and Rust APIs for building stateful data pipelines, a CLI for cluster management, a REST/gRPC API, a Kubernetes operator, and a Web UI.
- Key features: SQL planner, execution workers, state checkpointing, connectors to external systems.

##### High-Level Architecture Components
a. Clients
- CLI (“arroyo” binary, in crates/arroyo)
- Web UI (in webui/)
- Third-party clients using the REST/gRPC API (crates/arroyo-api, arroyo-rpc)
b. Control Plane
- API Server (crates/arroyo-api + arroyo-rpc + arroyo-openapi): exposes REST & gRPC endpoints, handles user requests, persists pipeline definitions in a relational store (PostgreSQL via migrations).
- Controller (crates/arroyo-controller): state machine managing job lifecycle, scheduling (embedded or Kubernetes scheduler), rescaling, checkpoint coordination.
- Kubernetes Operator (crates/arroyo-operator): integrates with k8s API to create & manage deployments of workers.
c. Data Plane
- Planner (crates/arroyo-planner): parses SQL, builds logical & physical execution plans.
- Workers (crates/arroyo-worker): execute dataflow operators (windowing, joins, UDFs), coordinate via network_manager.
- Connectors (crates/arroyo-connectors): source/sink adapters for Kafka, S3, filesystem, etc.
- UDF Host & Plugins (crates/arroyo-udf/*): run user-defined functions safely.
d. Supporting Services
- State Store (crates/arroyo-state): durable keyed state, checkpointing to external storage (crates/arroyo-storage).
- Metrics (crates/arroyo-metrics): collect & export runtime metrics.
- SQL Test Harness (crates/arroyo-sql-testing): validates SQL semantics (local testing only).
e. Infrastructure
- Relational database (PostgreSQL) for metadata & persistence
- Object store (e.g., AWS S3) for checkpoint/state snapshots
- Kafka / other brokers for streaming inputs
- Kubernetes cluster (optional) for containerized deployment
- Docker images (docker/) and Helm charts (k8s/) for deployment

##### Interactions & Data Flow
1. User invokes CLI or Web UI → API Server
→ API Server writes pipeline definitions to Postgres and returns status.
2. Controller watches Postgres metadata
→ Controller schedules/reschedules jobs by calling the Kubernetes Operator or embedded scheduler.
3. Kubernetes Operator creates Worker Pods
→ Pods spin up arroyo-worker processes.
4. Worker pulls execution plan from Controller (via RPC), then:
→ Planner compiles SQL to a physical plan.
→ Workers ingest from Connectors (Kafka, filesystem, etc.).
→ Data flows through operators, updating state in-memory.
→ Checkpoints are periodically written to Object Storage via arroyo-storage.
→ Sink connectors emit processed records back to external systems.
5. Metrics & logs are collected and exposed via Prometheus endpoints.

##### Architectural Patterns & Principles
- Microservices within a monorepo: separate crates for API, Controller, Planner, Worker, Connectors.
- Dataflow model: logical → physical plan, push-based streaming.
- State checkpointing for fault tolerance (exactly-once or at-least-once semantics).
- Pluggable connectors & UDFs for extensibility.
- Control/Data plane separation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients Layer
    subgraph "Clients"
        CLI["CLI"]:::clients
        WebUI["Web UI"]:::clients
        RESTClient["REST API Client"]:::clients
        RPCClient["gRPC API Client"]:::clients
    end

    %% Control Plane Layer
    subgraph "Control Plane"
        APIREST["API Server (REST)"]:::control
        APIGRPC["API Server (gRPC)"]:::control
        APIOpenAPI["API Server (OpenAPI)"]:::control
        Controller["Controller"]:::control
        Operator["Kubernetes Operator"]:::control
    end

    %% Data Plane Layer
    subgraph "Data Plane"
        Planner["Planner"]:::data
        Worker["Worker Process"]:::data
        Connectors["Connectors"]:::data
        subgraph "UDF Host & Plugins"
            UDFHost["UDF Host"]:::data
            UDFCommon["UDF Common Lib"]:::data
            UDFPlugin["UDF Plugin API"]:::data
            UDFPython["UDF Python Support"]:::data
            UDFMacros["UDF Procedural Macros"]:::data
        end
        StateStore["State Store"]:::data
        Storage["Storage (Checkpoint)"]:::data
        Metrics["Metrics"]:::data
        SQLTest["SQL Test Harness"]:::data
    end

    %% Infrastructure Layer
    subgraph "Infrastructure"
        Postgres["PostgreSQL"]:::infra
        ObjectStore["Object Store (S3)"]:::infra
        Kafka["Kafka / Brokers"]:::infra
        KubernetesCluster["Kubernetes Cluster"]:::infra
    end

    %% Data Flows & Control Flows
    CLI -->|invoke| APIREST
    WebUI -->|invoke| APIREST
    RESTClient -->|call REST| APIREST
    RPCClient -->|call gRPC| APIGRPC
    APIREST -->|write metadata| Postgres
    APIGRPC -->|write metadata| Postgres
    APIOpenAPI -->|generate docs| APIREST
    APIREST -->|enqueue job| Controller
    APIGRPC -->|enqueue job| Controller
    Controller -->|manage lifecycle| Operator
    Controller -->|read/write| Postgres
    Operator -->|deploy pods| KubernetesCluster
    Operator -->|launch| Worker
    Worker -->|request plan| Planner
    Planner -->|send plan| Worker
    Worker -->|ingest/emit| Kafka
    Worker -->|use| Connectors
    Worker -->|update state| StateStore
    Worker -->|checkpoint| Storage
    Worker -->|emit metrics| Metrics
    Worker -->|local test| SQLTest
    Worker -->|load| UDFHost
    UDFHost -->|use| UDFCommon
    UDFHost -->|load| UDFPlugin
    UDFHost -->|load| UDFPython
    UDFHost -->|use| UDFMacros

    %% Click Events
    click CLI "https://github.com/sudhishp/arroyo/blob/master/crates/arroyo/src/main.rs"
    click WebUI "https://github.com/sudhishp/arroyo/tree/master/webui/"
    click RESTClient "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-api/"
    click RPCClient "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-rpc/"
    click APIREST "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-api/src/"
    click APIGRPC "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-rpc/proto/"
    click APIOpenAPI "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-openapi/"
    click Controller "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-controller/src/"
    click Operator "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-operator/src/"
    click Planner "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-planner/src/"
    click Worker "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-worker/src/"
    click Connectors "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-connectors/src/"
    click UDFHost "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-host/src/"
    click UDFCommon "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-common/src/"
    click UDFPlugin "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-plugin/src/"
    click UDFPython "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-python/src/"
    click UDFMacros "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-udf/arroyo-udf-macros/src/"
    click StateStore "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-state/src/"
    click Storage "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-storage/src/"
    click Metrics "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-metrics/src/"
    click SQLTest "https://github.com/sudhishp/arroyo/tree/master/crates/arroyo-sql-testing/src/"

    %% Styles
    classDef clients fill:#D6C9FF,stroke:#8A2BE2,stroke-width:2px;
    classDef control fill:#CFE2FF,stroke:#0B5394,stroke-width:2px;
    classDef data fill:#D5E8D4,stroke:#448B00,stroke-width:2px;
    classDef infra fill:#FFE7BA,stroke:#994C00,stroke-width:2px;
```
