# apache/streampark architecture

> Architecture diagram of the GitHub repository apache/streampark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/streampark
- Repository: https://github.com/apache/streampark

## Overview

StreamPark is a platform and development framework for building and operating Flink and Spark streaming and batch applications across Standalone, YARN, and Kubernetes. A visible console workflow accepts build and progress requests, delegates them to engine-specific pipeline services, and returns pipeline status; the Flink controller can also expose Docker-resolution progress. The tree and README establish broader development, deployment, monitoring, and operations capabilities, but the excerpts do not show their full wiring, so those links are intentionally conservative.

Main entry points: [`streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java`](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java) (Flink pipeline API), [`streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java`](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java) (Spark pipeline API), [`streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java`](https://github.com/apache/streampark/blob/dev/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java) (Kubernetes session build), [`streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts`](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts) (Web console)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_console["Console API"]
  node_flink_api["Flink pipeline API"]
  node_spark_api["Spark pipeline API"]
  node_flink_assembler["Flink response mapper"]
  node_spark_assembler["Spark response mapper"]
  node_web_client["Web console<br/>[storageCache.ts]"]
end

subgraph group_pipeline["Build Pipelines"]
  node_flink_service["Flink build service"]
  node_spark_service["Spark build service"]
  node_pipeline_state["Pipeline state service"]
  node_pipeline_record["Build progress record"]
  node_packer["Flink build engine<br/>[BuildPipeline.java]"]
  node_k8s_session["Kubernetes session build"]
  node_maven["Maven packaging"]
  node_docker_progress["Docker build progress"]
end

subgraph group_shared["Shared Runtime"]
  node_database[("Console database")]
  node_flink_spark["Flink and Spark"]
  node_cluster["Compute environments"]
  node_common_config["Engine configuration<br/>[Configuration.java]"]
  node_common_fs["Filesystem operators<br/>[FsOperator.java]"]
end

node_user(("Platform user"))

node_user -->|"uses"| node_web_client
node_web_client -->|"requests"| node_flink_api
node_web_client -->|"requests"| node_spark_api
node_flink_api -->|"starts build"| node_flink_service
node_flink_api -->|"reads progress"| node_flink_service
node_spark_api -->|"starts build"| node_spark_service
node_spark_api -->|"reads progress"| node_spark_service
node_flink_api -->|"maps response"| node_flink_assembler
node_spark_api -->|"maps response"| node_spark_assembler
node_flink_api -->|"reads view"| node_pipeline_record
node_spark_api -->|"reads view"| node_pipeline_record
node_pipeline_state -->|"reads/writes"| node_database
node_pipeline_state -->|"manages records"| node_pipeline_record
node_k8s_session -->|"extends"| node_packer
node_k8s_session -->|"creates workspace"| node_common_fs
node_k8s_session -->|"builds shaded JAR"| node_maven
node_flink_api -->|"reads snapshot"| node_docker_progress
node_common_config -.->|"configures"| node_flink_spark
node_flink_spark -.->|"runs on"| node_cluster

click node_flink_api "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java"
click node_spark_api "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java"
click node_flink_service "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/FlinkApplicationBuildPipelineService.java"
click node_spark_service "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/SparkAplicationBuildPipelineService.java"
click node_pipeline_state "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/impl/AbstractApplicationBuildPipelineService.java"
click node_pipeline_record "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/entity/ApplicationBuildPipeline.java"
click node_flink_assembler "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/FlinkPipelineAssembler.java"
click node_spark_assembler "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/SparkPipelineAssembler.java"
click node_packer "https://github.com/apache/streampark/blob/dev/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/BuildPipeline.java"
click node_k8s_session "https://github.com/apache/streampark/blob/dev/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java"
click node_common_config "https://github.com/apache/streampark/blob/dev/streampark-common/src/main/java/org/apache/streampark/common/configuration/Configuration.java"
click node_common_fs "https://github.com/apache/streampark/blob/dev/streampark-common/src/main/java/org/apache/streampark/common/fs/FsOperator.java"
click node_web_client "https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_flink_api,node_spark_api,node_flink_assembler,node_spark_assembler,node_web_client,node_user toneBlue
class node_flink_service,node_spark_service,node_pipeline_state,node_pipeline_record,node_packer,node_k8s_session,node_maven,node_docker_progress toneAmber
class node_database,node_flink_spark,node_cluster,node_common_config,node_common_fs toneMint
```

## Components

### Console API

- [Flink pipeline API](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/FlinkPipelineController.java`
- [Spark pipeline API](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SparkPipelineController.java`
- [Flink response mapper](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/FlinkPipelineAssembler.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/FlinkPipelineAssembler.java`
- [Spark response mapper](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/SparkPipelineAssembler.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/assembler/SparkPipelineAssembler.java`
- [Web console](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts): `streampark-console/streampark-console-webapp/src/utils/cache/storageCache.ts`

### Build Pipelines

- [Flink build service](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/FlinkApplicationBuildPipelineService.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/FlinkApplicationBuildPipelineService.java`
- [Spark build service](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/SparkAplicationBuildPipelineService.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/SparkAplicationBuildPipelineService.java`
- [Pipeline state service](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/impl/AbstractApplicationBuildPipelineService.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/application/impl/AbstractApplicationBuildPipelineService.java`
- [Build progress record](https://github.com/apache/streampark/blob/dev/streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/entity/ApplicationBuildPipeline.java): `streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/entity/ApplicationBuildPipeline.java`
- [Flink build engine](https://github.com/apache/streampark/blob/dev/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/BuildPipeline.java): `streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/BuildPipeline.java`
- [Kubernetes session build](https://github.com/apache/streampark/blob/dev/streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java): `streampark-flink/streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline/impl/FlinkK8sSessionBuildPipeline.java`
- **Maven packaging**
- **Docker build progress**

### Shared Runtime

- **Console database**
- **Flink and Spark**
- **Compute environments**
- [Engine configuration](https://github.com/apache/streampark/blob/dev/streampark-common/src/main/java/org/apache/streampark/common/configuration/Configuration.java): `streampark-common/src/main/java/org/apache/streampark/common/configuration/Configuration.java`
- [Filesystem operators](https://github.com/apache/streampark/blob/dev/streampark-common/src/main/java/org/apache/streampark/common/fs/FsOperator.java): `streampark-common/src/main/java/org/apache/streampark/common/fs/FsOperator.java`

### Other components

- **Platform user**

## Connections

- Platform user → Web console: uses
- Web console → Flink pipeline API: requests
- Web console → Spark pipeline API: requests
- Flink pipeline API → Flink build service: starts build
- Flink pipeline API → Flink build service: reads progress
- Spark pipeline API → Spark build service: starts build
- Spark pipeline API → Spark build service: reads progress
- Flink pipeline API → Flink response mapper: maps response
- Spark pipeline API → Spark response mapper: maps response
- Flink pipeline API → Build progress record: reads view
- Spark pipeline API → Build progress record: reads view
- Pipeline state service → Console database: reads/writes
- Pipeline state service → Build progress record: manages records
- Kubernetes session build → Flink build engine: extends
- Kubernetes session build → Filesystem operators: creates workspace
- Kubernetes session build → Maven packaging: builds shaded JAR
- Flink pipeline API → Docker build progress: reads snapshot
- Engine configuration → Flink and Spark: configures
- Flink and Spark → Compute environments: runs on