# apache/flink architecture

> Architecture diagram of the GitHub repository apache/flink, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/flink
- Repository: https://github.com/apache/flink
- Video tour (about a minute): https://gitdiagram.com/apache/flink/video

## Overview

Apache Flink is a distributed stream- and batch-processing framework. A user defines a pipeline through Flink APIs, submits a packaged program through a client or application entry point, and Flink translates and executes it using runtime sources, operators, state, and sinks. Checkpointing and runtime coordination support fault tolerance; connectors integrate external data systems. The README also documents event-time processing, windows, back-pressure, and libraries. The tree and excerpts sample only selected modules, so several core execution stages are represented at module-directory level; arrows show documented workflow or behavior, not uninspected implementation calls.

Main entry points: [`flink-runtime/src/main/java/org/apache/flink/runtime`](https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime) (Flink runtime), [`flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java`](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java) (CLI frontend), [`flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java`](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java) (Packaged program), [`flink-clients/src/main/java/org/apache/flink/client/deployment/executors`](https://github.com/apache/flink/tree/master/flink-clients/src/main/java/org/apache/flink/client/deployment/executors) (Deployment executor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_submit["Submission"]
  node_cli["CLI frontend<br/>[CliFrontend.java]"]
  node_program["Packaged program"]
  node_translator["Pipeline translator"]
  node_executor["Deployment executor"]
end

subgraph group_execution["Data Processing"]
  node_runtime["Flink runtime"]
  node_sources["Sources"]
  node_operators["Stream operators"]
  node_sinks["Sinks"]
  node_connectors["File connector"]
end

subgraph group_runtime["Runtime Services"]
  node_checkpoint["Checkpoint storage"]
  node_changelog["State changelog"]
  node_leaders["Leader retrieval"]
  node_resources["Resource management"]
  node_shuffle["Tiered shuffle"]
end

subgraph group_access["Monitoring and Access"]
  node_metrics["Metric query"]
  node_dashboard["Web dashboard"]
  node_queryclient["Queryable state client"]
end

node_user(("Pipeline author"))
node_cluster(("Flink cluster"))
node_externaldata(("External data systems"))

node_user -->|"submits job"| node_cli
node_cli -->|"loads program"| node_program
node_program -->|"translates pipeline"| node_translator
node_cli -->|"selects deployment"| node_executor
node_executor -->|"deploys job"| node_cluster
node_cluster -->|"executes job"| node_runtime
node_runtime -->|"runs sources"| node_sources
node_runtime -->|"runs operators"| node_operators
node_runtime -->|"runs sinks"| node_sinks
node_sources -->|"emits records"| node_operators
node_operators -->|"forwards records"| node_sinks
node_connectors -->|"reads or writes"| node_externaldata
node_runtime -->|"persists checkpoints"| node_checkpoint
node_runtime -->|"manages changelog"| node_changelog
node_runtime -->|"retrieves leadership"| node_leaders
node_runtime -->|"manages resources"| node_resources
node_runtime -->|"transfers partitions"| node_shuffle
node_metrics -.->|"serves metrics"| node_dashboard
node_dashboard -->|"requests application data"| node_cluster
node_queryclient -.->|"queries managed state"| node_cluster

click node_cli "https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java"
click node_program "https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java"
click node_translator "https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/FlinkPipelineTranslator.java"
click node_executor "https://github.com/apache/flink/tree/master/flink-clients/src/main/java/org/apache/flink/client/deployment/executors"
click node_runtime "https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime"
click node_sources "https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source"
click node_operators "https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime"
click node_sinks "https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink"
click node_connectors "https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file"
click node_checkpoint "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/storage/JobManagerCheckpointStorage.java"
click node_changelog "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskExecutorStateChangelogStoragesManager.java"
click node_leaders "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/LeaderRetrievalService.java"
click node_resources "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManagerService.java"
click node_shuffle "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageNettyService.java"
click node_metrics "https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java"
click node_dashboard "https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/services/application.service.ts"
click node_queryclient "https://github.com/apache/flink/blob/master/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_program,node_translator,node_executor toneBlue
class node_runtime,node_sources,node_operators,node_sinks,node_connectors toneAmber
class node_checkpoint,node_changelog,node_leaders,node_resources,node_shuffle toneMint
class node_metrics,node_dashboard,node_queryclient toneRose
class node_user,node_cluster,node_externaldata toneIndigo
```

## Components

### Submission

- [CLI frontend](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java): `flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java`
- [Packaged program](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java): `flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java`
- [Pipeline translator](https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/FlinkPipelineTranslator.java): `flink-clients/src/main/java/org/apache/flink/client/FlinkPipelineTranslator.java`
- [Deployment executor](https://github.com/apache/flink/tree/master/flink-clients/src/main/java/org/apache/flink/client/deployment/executors): `flink-clients/src/main/java/org/apache/flink/client/deployment/executors`

### Data Processing

- [Flink runtime](https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime): `flink-runtime/src/main/java/org/apache/flink/runtime`
- [Sources](https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source): `flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source`
- [Stream operators](https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime): `flink-runtime/src/main/java/org/apache/flink/runtime`
- [Sinks](https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink): `flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink`
- [File connector](https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file): `flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file`

### Runtime Services

- [Checkpoint storage](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/storage/JobManagerCheckpointStorage.java): `flink-runtime/src/main/java/org/apache/flink/runtime/state/storage/JobManagerCheckpointStorage.java`
- [State changelog](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskExecutorStateChangelogStoragesManager.java): `flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskExecutorStateChangelogStoragesManager.java`
- [Leader retrieval](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/LeaderRetrievalService.java): `flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/LeaderRetrievalService.java`
- [Resource management](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManagerService.java): `flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManagerService.java`
- [Tiered shuffle](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageNettyService.java): `flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageNettyService.java`

### Monitoring and Access

- [Metric query](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java): `flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java`
- [Web dashboard](https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/services/application.service.ts): `flink-runtime-web/web-dashboard/src/app/services/application.service.ts`
- [Queryable state client](https://github.com/apache/flink/blob/master/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java): `flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java`

### Other components

- **Pipeline author**
- **Flink cluster**
- **External data systems**

## Connections

- Pipeline author → CLI frontend: submits job
- CLI frontend → Packaged program: loads program
- Packaged program → Pipeline translator: translates pipeline
- CLI frontend → Deployment executor: selects deployment
- Deployment executor → Flink cluster: deploys job
- Flink cluster → Flink runtime: executes job
- Flink runtime → Sources: runs sources
- Flink runtime → Stream operators: runs operators
- Flink runtime → Sinks: runs sinks
- Sources → Stream operators: emits records
- Stream operators → Sinks: forwards records
- File connector → External data systems: reads or writes
- Flink runtime → Checkpoint storage: persists checkpoints
- Flink runtime → State changelog: manages changelog
- Flink runtime → Leader retrieval: retrieves leadership
- Flink runtime → Resource management: manages resources
- Flink runtime → Tiered shuffle: transfers partitions
- Metric query → Web dashboard: serves metrics
- Web dashboard → Flink cluster: requests application data
- Queryable state client → Flink cluster: queries managed state