# estuary/flow architecture

> Architecture diagram of the GitHub repository estuary/flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/estuary/flow
- Repository: https://github.com/estuary/flow

## Overview

##### High-Level Architecture Components

The system should be divided into these main components:

a) Control Plane:
- Agent service (crates/agent)
- Data plane controller (crates/data-plane-controller)
- Authentication/Authorization services
- Configuration management

b) Data Plane:
- Connectors
- Materialization connectors
- Derivation engine
- Collection storage
- Gazette streaming broker

c) Client Interfaces:
- Web Application (dashboard)
- CLI (flowctl)
- API endpoints

##### Key Relationships and Data Flow

**[Client Interfaces] → [Control Plane] → [Data Plane]**
- ↓
- [Storage Layer]

With detailed flows:
**[Capture Sources] → [Collections] → [Derivations] → [Materializations]**
- ↑↓
- [Cloud Storage]

##### Component Details

For each major component, include:

a) Control Plane:
- Agent API endpoints (crates/agent/src/api)
- Controller management (crates/data-plane-controller/src/controller.rs)
- Task scheduling and monitoring
- Configuration management

b) Data Plane:
- Connector management system
- Runtime execution engine (crates/runtime)
- Transaction processing
- State management (RocksDB integration)

c) Storage Layer:
- Collection storage in cloud
- Gazette broker integration
- State storage (RocksDB)

##### Technical Implementation Details

Technical aspects:

- Rust/Go hybrid architecture
- CGO service channel for performance
- RocksDB for state management
- Cloud storage integration
- Gazette streaming broker

##### Specific Implementation Notes

Technical details about:
- Optimistic pipelining
- In-memory reduction grouping
- Register co-location
- Vectorized operations
- Zero-copy operations

##### External Integration Points

Connections to:
- Cloud storage providers
- Database systems
- Message queues
- SaaS platforms
- Custom connectors

- Millisecond-latency processing
- Unified batch/streaming paradigm
- Scalable data processing
- State management
- Connector ecosystem

This system design should capture both the high-level architecture and the detailed implementation specifics that make Flow unique, while remaining clear and understandable to technical stakeholders.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client
        CLI["CLI (flowctl)"]:::client
        WebApp["Web Application"]:::client
        API["API Endpoints"]:::client
    end

    subgraph ControlPlane
        Agent["Agent Service"]:::control
        DPC["Data Plane Controller"]:::control
        Auth["Authorization System"]:::control
        Config["Configuration Management"]:::control
        TaskSched["Task Scheduling"]:::control
        Proto["Protocol Definitions"]:::control
    end

    subgraph DataPlane
        Capture["Capture System"]:::data
        Material["Materialization System"]:::data
        Derive["Derivation Engine"]:::data
        Runtime["Runtime Engine"]:::data
    end

    subgraph StorageLayer
        CollStore["Collection Storage"]:::storage
        Gazette["Gazette Streaming Broker"]:::storage
        RocksDB["RocksDB State Store"]:::storage
        CloudStore["Cloud Storage"]:::storage
    end

    subgraph ExternalSystems
        DB[(Databases)]:::external
        MQ[("Message Queues")]:::external
        SaaS["SaaS Platforms"]:::external
        Custom["Custom Connectors"]:::external
    end

    %% Client to Control Plane
    CLI --> Agent
    WebApp --> Agent
    API --> Agent

    %% Control Plane Internal
    Agent --> DPC
    Agent --> Auth
    Agent --> Config
    DPC --> TaskSched
    DPC --> Proto

    %% Control to Data Plane
    DPC --> Runtime
    TaskSched --> Runtime

    %% Data Plane Internal
    Runtime --> Capture
    Runtime --> Material
    Runtime --> Derive

    %% Storage Connections
    Derive --> CollStore
    Capture --> Gazette
    Material --> Gazette
    Runtime --> RocksDB
    CollStore --> CloudStore

    %% External Connections
    Capture --> ExternalSystems
    Material --> ExternalSystems

    %% Styles
    classDef control fill:#b3d1ff,stroke:#4d88ff
    classDef data fill:#b3ffb3,stroke:#33cc33
    classDef storage fill:#fff2b3,stroke:#ffdb4d
    classDef external fill:#ffb3b3,stroke:#ff4d4d
    classDef client fill:#e6b3ff,stroke:#9933ff

    %% Click Events
    click Agent "https://github.com/estuary/flow/tree/master/crates/agent/"
    click DPC "https://github.com/estuary/flow/tree/master/crates/data-plane-controller/"
    click Auth "https://github.com/estuary/flow/tree/master/crates/agent/src/api/authorize_*"
    click Capture "https://github.com/estuary/flow/tree/master/crates/runtime/src/capture/"
    click Material "https://github.com/estuary/flow/tree/master/crates/runtime/src/materialize/"
    click Derive "https://github.com/estuary/flow/tree/master/crates/runtime/src/derive/"
    click Gazette "https://github.com/estuary/flow/tree/master/crates/gazette/"
    click CLI "https://github.com/estuary/flow/tree/master/crates/flowctl/"
    click WebApp "https://github.com/estuary/flow/tree/master/crates/flow-web/"
    click Runtime "https://github.com/estuary/flow/tree/master/crates/runtime/"
    click Proto "https://github.com/estuary/flow/tree/master/crates/proto-flow/"
    click CollStore "https://github.com/estuary/flow/tree/master/crates/doc/"
    click Config "https://github.com/estuary/flow/tree/master/crates/models/"
    click TaskSched "https://github.com/estuary/flow/blob/master/crates/runtime/src/task_service.rs"
```
