How apache/flink-ml works
Project type and purpose
• This is a multi-module, Maven-based library suite (not a stand-alone app) providing machine learning APIs and infrastructure on top of Apache Flink.
• It supports both Java and Python frontends (flink-ml-core / flink-ml-lib for Java, flink-ml-python for Python), iteration primitives, “servable” deployment APIs, benchmark tooling, examples, distribution packaging, and documentation.
File structure
• Top-level modules (each with its own pom.xml):
• flink-ml-core: common API, pipeline builder, Graph abstractions, core utilities
• flink-ml-lib: concrete ML algorithms (classification, clustering, feature transformers, evaluation, etc.)
• flink-ml-iteration & flink-ml-iteration-common: streaming iteration primitives and coordination layer
• flink-ml-servable-core & flink-ml-servable-lib: runtime for exporting and serving models (DataFrame, ModelServable, Builder)
• flink-ml-python: Python wrapper, APIs, examples, docs, packaging scripts
• flink-ml-dist / flink-ml-uber: assembly of binaries, CLI scripts for running and packaging
• flink-ml-benchmark: performance bench tooling and data generators
• flink-ml-examples / flink-ml-tests: sample pipelines and end-to-end tests
• docs/: site source (Hugo) and styling
• CI workflows under .github/workflows; scripts under tools/; top-level README describes usage.
README
• Project description: “library providing ML APIs & infrastructures for Flink pipelines.”
• Quick-starts for Java/Python, build instructions, benchmark pointer, docs location.
a. Main components
• User layer: Java client, Python client
• API layer: flink-ml-core (AlgoOperator, Estimator, Transformer, Pipeline/Graph)
• Algorithm implementations: flink-ml-lib (knn, logistic regression, kmeans, feature ops, evaluators)
• Iteration framework: flink-ml-iteration modules providing broadcast, feedback channels, checkpoint alignment
• Servable layer: flink-ml-servable-core & lib exposing DataFrame, ModelServable, TransformerServable for serving
• Distribution & packaging: flink-ml-dist, flink-ml-uber, CLI scripts, Python setup.py
• Benchmarking: flink-ml-benchmark with generators, runner, result visualizer
• Examples & docs: flink-ml-examples, docs/ site
• External dependency: Apache Flink runtime cluster
b. Relationships & interactions
• Java/Python client → calls API layer in flink-ml-core (or pyflink.ml)
• API layer → builds a Pipeline/Graph that references operators from algorithm modules
• GraphExecutionHelper → uses Flink DataStream API + iteration components to orchestrate streaming jobs
• Iteration modules → provide control paths for per-round or all-round iterations under the hood
• When serving: PipelineModelServable (servable-core) wraps the pipeline logic and exposes a lightweight runtime
• Distribution modules package all needed jars & Python artifacts into downloadable bundles
• Benchmark module drives the API layer with synthetic data, reports results to scripts/visualizer
• Examples demonstrate end-to-end flows from client-API through pipeline building to Flink job submission
• Documentation site lives under docs/, consumed by Hugo to produce HTML; linked from GitHub pages
c. Architectural patterns & design principles
• Modular, Maven multi-module layering
• Separation of concerns: API vs. implementations vs. iteration vs. serving vs. packaging
• Builder/Graph pattern for pipeline composition
• Use of factories and wrapper operators for broadcast & feedback iteration
• Dual language support: Java APIs in flink-ml-core vs. Python wrappers in flink-ml-python
• CI/CD: GitHub Actions for Java and Python tests
d. Key technologies & frameworks
• Apache Flink DataStream API
• Maven multi-module build
• Hugo static site generator (docs/)
• Python setuptools (setup.py)
• GitHub Actions CI
Elements
• User → API → Implementation → Flink runtime
• API → Iteration primitives
• PipelineModel → Servable runtime
• Build scripts → assembled distribution
• Rectangles for code modules