How alibaba/fluss works
Project’s Nature and Main Purpose
• Fluss is a streaming storage system designed for real-time analytics and optimized for lakehouse architectures. It acts as an ingestion and storage layer that bridges real-time data streaming and data lake storage. It integrates with popular compute engines such as Apache Flink (with Apache Spark and StarRocks planned).
Break Down the Key Architectural Layers and Modules
A. Client Side (Data Producers / Client Library)
– Represent the fluss-client module, which supplies API classes (e.g., Connection, Lookup functions, table write/scan operations, token management, etc.). This layer is your entry point for producers to feed streaming data.
B. Communication and Gateway Layer
C. Server / Coordinator Cluster
• The coordinator/sub-coordinator components (for managing metadata, coordination, tablet assignment, state transitions and leader-election logic).
• Partition/tablet management, log management (including snapshot and replication management), and the KV storage submodules.
• Components related to replication, delayed fetches, snapshots, and local storage operations.
D. Underlying Storage and Filesystem Abstraction
– Illustrate the fluss-common module as the shared library for annotations, configurations, bucketing, file system abstractions, and memory management. Also, include the fluss-filesystems modules which contain integrations for various distributed storage systems such as HDFS, OSS, and S3.
E. Compute Engine Integration
– Use the fluss-flink module to show how Fluss plugs into Apache Flink (and note that integration with Spark/StarRocks is on the roadmap). This connector ensures that streaming storage is available to processing frameworks.
F. Additional Supporting Modules and Tools
• fluss-dist: Represents the packaged distribution with launch scripts and configuration files.
• fluss-protogen: Shows the code generation for protocol buffers and message definitions, ensuring consistency in the API.
• Website and Documentation: While not a runtime component, the website folder shows the documentation, blog posts, and guides which support developer and user adoption.
Interactions and Data Flow
• “Client API” (from fluss-client) communicating with the “Fluss Server” through the RPC layer (fluss-rpc).
• Within the server cluster, show the coordination and management flow:
– Inbound write/read requests are handled by a combination of the Coordinator, Tablets, KV storage mechanisms, snapshot and log management processes.
• Integration endpoints to Apache Flink (and note potential integrations for Spark/StarRocks) that consume the streaming data for analytics.
Important Architectural Patterns and Principles
• Layered Architecture: Differentiate clearly between the client, communication, server (coordinator), storage, and compute integration layers.
• Modularity: Each major functional block (e.g., fluss-client, fluss-server, fluss-rpc, file system integrations) should be represented as separate, yet interconnected components.
• Scalability & Replication: Indicate how the server component handles partitioning, tablet management, and replication; be sure to show any coordination (via ZooKeeper or similar coordination protocols if applicable, as hinted by the presence of zk directories within fluss-server).
• Observability: Include the metrics reporting subsystems (for example, JMX/Prometheus) as key monitoring components.
• Modular and layered design so that any observer can follow the data’s journey from ingestion to storage and analytics.