# alibaba/fluss architecture

> Architecture diagram of the GitHub repository alibaba/fluss, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba/fluss
- Repository: https://github.com/alibaba/fluss

## Overview

##### Project’s Nature and Main Purpose
- Fluss is a streaming storage system designed for real-time analytics and optimized for lakehouse architectures. It acts as an ingestion and storage layer that bridges real-time data streaming and data lake storage. It integrates with popular compute engines such as Apache Flink (with Apache Spark and StarRocks planned).

##### Break Down the Key Architectural Layers and Modules
A. Client Side (Data Producers / Client Library)
– Represent the fluss-client module, which supplies API classes (e.g., Connection, Lookup functions, table write/scan operations, token management, etc.). This layer is your entry point for producers to feed streaming data.
B. Communication and Gateway Layer
C. Server / Coordinator Cluster
- The coordinator/sub-coordinator components (for managing metadata, coordination, tablet assignment, state transitions and leader-election logic).
- Partition/tablet management, log management (including snapshot and replication management), and the KV storage submodules.
- Components related to replication, delayed fetches, snapshots, and local storage operations.
D. Underlying Storage and Filesystem Abstraction
– Illustrate the fluss-common module as the shared library for annotations, configurations, bucketing, file system abstractions, and memory management. Also, include the fluss-filesystems modules which contain integrations for various distributed storage systems such as HDFS, OSS, and S3.
E. Compute Engine Integration
– Use the fluss-flink module to show how Fluss plugs into Apache Flink (and note that integration with Spark/StarRocks is on the roadmap). This connector ensures that streaming storage is available to processing frameworks.
F. Additional Supporting Modules and Tools
- fluss-dist: Represents the packaged distribution with launch scripts and configuration files.
- fluss-protogen: Shows the code generation for protocol buffers and message definitions, ensuring consistency in the API.
- Website and Documentation: While not a runtime component, the website folder shows the documentation, blog posts, and guides which support developer and user adoption.

##### Interactions and Data Flow
- “Client API” (from fluss-client) communicating with the “Fluss Server” through the RPC layer (fluss-rpc).
- Within the server cluster, show the coordination and management flow:
– Inbound write/read requests are handled by a combination of the Coordinator, Tablets, KV storage mechanisms, snapshot and log management processes.
- Integration endpoints to Apache Flink (and note potential integrations for Spark/StarRocks) that consume the streaming data for analytics.

##### Important Architectural Patterns and Principles
- Layered Architecture: Differentiate clearly between the client, communication, server (coordinator), storage, and compute integration layers.
- Modularity: Each major functional block (e.g., fluss-client, fluss-server, fluss-rpc, file system integrations) should be represented as separate, yet interconnected components.
- Scalability & Replication: Indicate how the server component handles partitioning, tablet management, and replication; be sure to show any coordination (via ZooKeeper or similar coordination protocols if applicable, as hinted by the presence of zk directories within fluss-server).
- Observability: Include the metrics reporting subsystems (for example, JMX/Prometheus) as key monitoring components.

- Modular and layered design so that any observer can follow the data’s journey from ingestion to storage and analytics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        DP["Data Producer Application"] 
        CLIENT["Fluss Client API"]:::client
        KAFKA["Kafka Integration Module"]:::client
    end

    %% Communication Layer
    subgraph "Communication Layer"
        RPC["RPC Communication (Netty-based)"]:::rpc
    end

    %% Server/Coordinator Layer
    subgraph "Server/Coordinator Layer"
        SERVER["Fluss Server Cluster"]:::server
    end

    %% Storage and Compute Integration Layer
    subgraph "Storage/Integration Layer"
        STORAGE["Filesystem Abstractions (HDFS/OSS/S3)"]:::storage
        COMPUTE["Flink Connector (Apache Flink)"]:::compute
    end

    %% Supporting Modules and Tools
    subgraph "Supporting Modules"
        PROTOGEN["Protocol Buffers Codegen"]:::supporting
        DIST["Packaged Distribution & Launch Scripts"]:::supporting
        METRICSJMX["JMX Reporter"]:::metrics
        METRICSPROM["Prometheus Reporter"]:::metrics
    end

    %% Data Flow Connections
    DP -->|"feeds"| CLIENT
    CLIENT -->|"calls"| RPC
    RPC -->|"communicates with"| SERVER
    SERVER -->|"manages data via"| STORAGE
    SERVER -->|"integrates with"| COMPUTE
    CLIENT -->|"optional integration"| KAFKA
    SERVER -->|"triggers"| METRICSJMX
    SERVER -->|"triggers"| METRICSPROM
    SERVER -->|"supported by"| PROTOGEN
    SERVER -->|"supported by"| DIST

    %% Click Events
    click CLIENT "https://github.com/alibaba/fluss/tree/main/fluss-client/"
    click STORAGE "https://github.com/alibaba/fluss/tree/main/fluss-common/"
    click RPC "https://github.com/alibaba/fluss/tree/main/fluss-rpc/"
    click SERVER "https://github.com/alibaba/fluss/tree/main/fluss-server/"
    click DIST "https://github.com/alibaba/fluss/tree/main/fluss-dist/"
    click COMPUTE "https://github.com/alibaba/fluss/tree/main/fluss-flink/"
    click KAFKA "https://github.com/alibaba/fluss/tree/main/fluss-kafka/"
    click PROTOGEN "https://github.com/alibaba/fluss/tree/main/fluss-protogen/"
    click METRICSJMX "https://github.com/alibaba/fluss/tree/main/fluss-metrics/fluss-metrics-jmx/"
    click METRICSPROM "https://github.com/alibaba/fluss/tree/main/fluss-metrics/fluss-metrics-prometheus/"

    %% Styles
    classDef client fill:#a3d3ff,stroke:#333,stroke-width:1px;
    classDef rpc fill:#ffcc99,stroke:#333,stroke-width:1px;
    classDef server fill:#d5ffa3,stroke:#333,stroke-width:1px;
    classDef storage fill:#f7a1c4,stroke:#333,stroke-width:1px;
    classDef compute fill:#ffb3e6,stroke:#333,stroke-width:1px;
    classDef supporting fill:#ddccff,stroke:#333,stroke-width:1px;
    classDef metrics fill:#c2f0c2,stroke:#333,stroke-width:1px;
```
