How sudhishp/arroyo works
Project Type and Purpose
• This is a distributed, real-time stream processing engine (Arroyo) written in Rust.
• It provides SQL and Rust APIs for building stateful data pipelines, a CLI for cluster management, a REST/gRPC API, a Kubernetes operator, and a Web UI.
• Key features: SQL planner, execution workers, state checkpointing, connectors to external systems.
High-Level Architecture Components
a. Clients
• CLI (“arroyo” binary, in crates/arroyo)
• Web UI (in webui/)
• Third-party clients using the REST/gRPC API (crates/arroyo-api, arroyo-rpc)
b. Control Plane
• API Server (crates/arroyo-api + arroyo-rpc + arroyo-openapi): exposes REST & gRPC endpoints, handles user requests, persists pipeline definitions in a relational store (PostgreSQL via migrations).
• Controller (crates/arroyo-controller): state machine managing job lifecycle, scheduling (embedded or Kubernetes scheduler), rescaling, checkpoint coordination.
• Kubernetes Operator (crates/arroyo-operator): integrates with k8s API to create & manage deployments of workers.
c. Data Plane
• Planner (crates/arroyo-planner): parses SQL, builds logical & physical execution plans.
• Workers (crates/arroyo-worker): execute dataflow operators (windowing, joins, UDFs), coordinate via network_manager.
• Connectors (crates/arroyo-connectors): source/sink adapters for Kafka, S3, filesystem, etc.
• UDF Host & Plugins (crates/arroyo-udf/*): run user-defined functions safely.
d. Supporting Services
• State Store (crates/arroyo-state): durable keyed state, checkpointing to external storage (crates/arroyo-storage).
• Metrics (crates/arroyo-metrics): collect & export runtime metrics.
• SQL Test Harness (crates/arroyo-sql-testing): validates SQL semantics (local testing only).
e. Infrastructure
• Relational database (PostgreSQL) for metadata & persistence
• Object store (e.g., AWS S3) for checkpoint/state snapshots
• Kafka / other brokers for streaming inputs
• Kubernetes cluster (optional) for containerized deployment
• Docker images (docker/) and Helm charts (k8s/) for deployment
Interactions & Data Flow
1. User invokes CLI or Web UI → API Server
→ API Server writes pipeline definitions to Postgres and returns status.
2. Controller watches Postgres metadata
→ Controller schedules/reschedules jobs by calling the Kubernetes Operator or embedded scheduler.
3. Kubernetes Operator creates Worker Pods
→ Pods spin up arroyo-worker processes.
4. Worker pulls execution plan from Controller (via RPC), then:
→ Planner compiles SQL to a physical plan.
→ Workers ingest from Connectors (Kafka, filesystem, etc.).
→ Data flows through operators, updating state in-memory.
→ Checkpoints are periodically written to Object Storage via arroyo-storage.
→ Sink connectors emit processed records back to external systems.
5. Metrics & logs are collected and exposed via Prometheus endpoints.
Architectural Patterns & Principles
• Microservices within a monorepo: separate crates for API, Controller, Planner, Worker, Connectors.
• Dataflow model: logical → physical plan, push-based streaming.
• State checkpointing for fault tolerance (exactly-once or at-least-once semantics).
• Pluggable connectors & UDFs for extensibility.
• Control/Data plane separation.