# apache/flink-ml architecture

> Architecture diagram of the GitHub repository apache/flink-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/flink-ml
- Repository: https://github.com/apache/flink-ml

## Overview

##### Project type and purpose
- This is a multi-module, Maven-based library suite (not a stand-alone app) providing machine learning APIs and infrastructure on top of Apache Flink.
- It supports both Java and Python frontends (flink-ml-core / flink-ml-lib for Java, flink-ml-python for Python), iteration primitives, “servable” deployment APIs, benchmark tooling, examples, distribution packaging, and documentation.

##### File structure
- Top-level modules (each with its own pom.xml):
- flink-ml-core: common API, pipeline builder, Graph abstractions, core utilities
- flink-ml-lib: concrete ML algorithms (classification, clustering, feature transformers, evaluation, etc.)
- flink-ml-iteration & flink-ml-iteration-common: streaming iteration primitives and coordination layer
- flink-ml-servable-core & flink-ml-servable-lib: runtime for exporting and serving models (DataFrame, ModelServable, Builder)
- flink-ml-python: Python wrapper, APIs, examples, docs, packaging scripts
- flink-ml-dist / flink-ml-uber: assembly of binaries, CLI scripts for running and packaging
- flink-ml-benchmark: performance bench tooling and data generators
- flink-ml-examples / flink-ml-tests: sample pipelines and end-to-end tests
- docs/: site source (Hugo) and styling
- CI workflows under .github/workflows; scripts under tools/; top-level README describes usage.

##### README
- Project description: “library providing ML APIs & infrastructures for Flink pipelines.”
- Quick-starts for Java/Python, build instructions, benchmark pointer, docs location.

a. Main components
- User layer: Java client, Python client
- API layer: flink-ml-core (AlgoOperator, Estimator, Transformer, Pipeline/Graph)
- Algorithm implementations: flink-ml-lib (knn, logistic regression, kmeans, feature ops, evaluators)
- Iteration framework: flink-ml-iteration modules providing broadcast, feedback channels, checkpoint alignment
- Servable layer: flink-ml-servable-core & lib exposing DataFrame, ModelServable, TransformerServable for serving
- Distribution & packaging: flink-ml-dist, flink-ml-uber, CLI scripts, Python setup.py
- Benchmarking: flink-ml-benchmark with generators, runner, result visualizer
- Examples & docs: flink-ml-examples, docs/ site
- External dependency: Apache Flink runtime cluster

b. Relationships & interactions
- Java/Python client → calls API layer in flink-ml-core (or pyflink.ml)
- API layer → builds a Pipeline/Graph that references operators from algorithm modules
- GraphExecutionHelper → uses Flink DataStream API + iteration components to orchestrate streaming jobs
- Iteration modules → provide control paths for per-round or all-round iterations under the hood
- When serving: PipelineModelServable (servable-core) wraps the pipeline logic and exposes a lightweight runtime
- Distribution modules package all needed jars & Python artifacts into downloadable bundles
- Benchmark module drives the API layer with synthetic data, reports results to scripts/visualizer
- Examples demonstrate end-to-end flows from client-API through pipeline building to Flink job submission
- Documentation site lives under docs/, consumed by Hugo to produce HTML; linked from GitHub pages

c. Architectural patterns & design principles
- Modular, Maven multi-module layering
- Separation of concerns: API vs. implementations vs. iteration vs. serving vs. packaging
- Builder/Graph pattern for pipeline composition
- Use of factories and wrapper operators for broadcast & feedback iteration
- Dual language support: Java APIs in flink-ml-core vs. Python wrappers in flink-ml-python
- CI/CD: GitHub Actions for Java and Python tests

d. Key technologies & frameworks
- Apache Flink DataStream API
- Maven multi-module build
- Hugo static site generator (docs/)
- Python setuptools (setup.py)
- GitHub Actions CI

##### Elements
- User → API → Implementation → Flink runtime
- API → Iteration primitives
- PipelineModel → Servable runtime
- Build scripts → assembled distribution
- Rectangles for code modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    subgraph "User Layer"
        JavaClient["Java Client"]:::client
        PythonClient["Python Client"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        CoreAPI["Core API (Pipeline/Graph)"]:::api
    end

    %% Algorithm Implementations
    subgraph "Algorithm Implementations"
        AlgoImpl["Algorithms Module"]:::impl
    end

    %% Iteration Framework
    subgraph "Iteration Framework"
        IterVer["Iteration 1.15 Module"]:::iteration
        IterCommon["Iteration Common Module"]:::iteration
    end

    %% Servable Layer
    subgraph "Servable Layer"
        ServableCore["Servable Runtime Core"]:::servable
        ServableLib["Servable Implementations"]:::servable
    end

    %% Distribution & Packaging
    subgraph "Distribution & Packaging"
        DistBin["Dist & CLI Scripts"]:::pkg
        UberJar["Uber JAR Assembly"]:::pkg
        PyPkg["Python Packaging"]:::pkg
    end

    %% Benchmarking
    subgraph "Benchmarking"
        Benchmark["Benchmark Tooling"]:::benchmark
    end

    %% Examples & Tests
    subgraph "Examples & Tests"
        Examples["Examples"]:::examples
        Tests["Tests"]:::examples
    end

    %% Documentation
    Docs["Documentation Site"]:::docs

    %% External System
    Flink["Apache Flink Cluster"]:::external

    %% Connections
    JavaClient -->|uses| CoreAPI
    PythonClient -->|uses| CoreAPI
    CoreAPI -->|invokes| AlgoImpl
    CoreAPI -->|uses| IterVer
    CoreAPI -->|uses| IterCommon
    CoreAPI -->|builds| ServableCore
    ServableCore -->|wraps| ServableLib
    CoreAPI -->|executes via| Flink
    ServableCore -->|deploys on| Flink
    DistBin -->|packages| UberJar
    DistBin -->|packages| PyPkg
    Benchmark -->|drives| CoreAPI
    Benchmark -->|runs on| Flink
    Examples -->|demo| CoreAPI
    Examples -->|submit to| Flink
    Tests -->|validate| CoreAPI
    Docs -->|generates site| Docs

    %% Click Events
    click JavaClient "https://github.com/apache/flink-ml/tree/master/flink-ml-core/src/main/java/org/apache/flink/ml/api"
    click PythonClient "https://github.com/apache/flink-ml/blob/master/flink-ml-python/pyflink/ml/api.py"
    click PythonClient "https://github.com/apache/flink-ml/blob/master/flink-ml-python/pyflink/ml/builder.py"
    click CoreAPI "https://github.com/apache/flink-ml/tree/master/flink-ml-core/src/main/java/org/apache/flink/ml/builder"
    click AlgoImpl "https://github.com/apache/flink-ml/tree/master/flink-ml-lib/src/main/java/org/apache/flink/ml"
    click IterVer "https://github.com/apache/flink-ml/tree/master/flink-ml-iteration/flink-ml-iteration-1.15/src/main/java/org/apache/flink/iteration"
    click IterCommon "https://github.com/apache/flink-ml/tree/master/flink-ml-iteration-common/src/main/java/org/apache/flink/iteration"
    click ServableCore "https://github.com/apache/flink-ml/tree/master/flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable"
    click ServableLib "https://github.com/apache/flink-ml/tree/master/flink-ml-servable-lib/src/main/java/org/apache/flink/ml"
    click DistBin "https://github.com/apache/flink-ml/tree/master/flink-ml-dist/src/main/assemblies"
    click UberJar "https://github.com/apache/flink-ml/tree/master/flink-ml-uber/src/main/resources/META-INF/licenses"
    click PyPkg "https://github.com/apache/flink-ml/blob/master/flink-ml-python/setup.py"
    click Benchmark "https://github.com/apache/flink-ml/tree/master/flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark"
    click Examples "https://github.com/apache/flink-ml/tree/master/flink-ml-examples/src/main/java/org/apache/flink/ml/examples"
    click Docs "https://github.com/apache/flink-ml/tree/master/docs/"

    %% Styles
    classDef client fill:#AEDFF7,stroke:#0B5394,color:#000
    classDef api fill:#D5E8D4,stroke:#82B366,color:#000
    classDef impl fill:#F8CECC,stroke:#B85450,color:#000
    classDef iteration fill:#F4CCCC,stroke:#A61C00,color:#000
    classDef servable fill:#E1D5E7,stroke:#9673A6,color:#000
    classDef pkg fill:#D9D9D9,stroke:#7F7F7F,color:#000
    classDef benchmark fill:#FFF2CC,stroke:#BF9000,color:#000
    classDef examples fill:#EAF1DD,stroke:#6AA84F,color:#000
    classDef docs fill:#FFE599,stroke:#B45F06,color:#000
    classDef external fill:#B7CEEC,stroke:#3C78D8,color:#000
```
