How apache/streampark works
StreamPark is a platform and development framework for building and operating Flink and Spark streaming and batch applications across Standalone, YARN, and Kubernetes. A visible console workflow accepts build and progress requests, delegates them to engine-specific pipeline services, and returns pipeline status; the Flink controller can also expose Docker-resolution progress. The tree and README establish broader development, deployment, monitoring, and operations capabilities, but the excerpts do not show their full wiring, so those links are intentionally conservative.
Main entry points: FlinkPipelineController.java Flink pipeline API · SparkPipelineController.java Spark pipeline API · FlinkK8sSessionBuildPipeline.java Kubernetes session build · storageCache.ts Web console
Components 19
Console API
- Flink pipeline API
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java - Spark pipeline API
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java - Flink response mapper
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/FlinkPipelineAssembler.java - Spark response mapper
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/SparkPipelineAssembler.java - Web console
streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts
Build Pipelines
- Flink build service
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/FlinkApplicationBuildPipelineService.java - Spark build service
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/SparkAplicationBuildPipelineService.java - Pipeline state service
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/impl/AbstractApplicationBuildPipelineService.java - Build progress record
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/entity/ApplicationBuildPipeline.java - Flink build engine
streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/BuildPipeline.java - Kubernetes session build
streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java - Maven packaging
- Docker build progress
Shared Runtime
- Console database
- Flink and Spark
- Compute environments
- Engine configuration
streampark-common/src/main/java/org/apache/streampark/common/configuration/Configuration.java - Filesystem operators
streampark-common/src/main/java/org/apache/streampark/common/fs/FsOperator.java
Other components
- Platform user
Connections 19
- Platform user → Web console: uses
- Web console → Flink pipeline API: requests
- Web console → Spark pipeline API: requests
- Flink pipeline API → Flink build service: starts build
- Flink pipeline API → Flink build service: reads progress
- Spark pipeline API → Spark build service: starts build
- Spark pipeline API → Spark build service: reads progress
- Flink pipeline API → Flink response mapper: maps response
- Spark pipeline API → Spark response mapper: maps response
- Flink pipeline API → Build progress record: reads view
- Spark pipeline API → Build progress record: reads view
- Pipeline state service → Console database: reads/writes
- Pipeline state service → Build progress record: manages records
- Kubernetes session build → Flink build engine: extends
- Kubernetes session build → Filesystem operators: creates workspace
- Kubernetes session build → Maven packaging: builds shaded JAR
- Flink pipeline API → Docker build progress: reads snapshot
- Engine configuration → Flink and Spark: configures
- Flink and Spark → Compute environments: runs on