How apache/flink works
Apache Flink is a distributed stream- and batch-processing framework. A user defines a pipeline through Flink APIs, submits a packaged program through a client or application entry point, and Flink translates and executes it using runtime sources, operators, state, and sinks. Checkpointing and runtime coordination support fault tolerance; connectors integrate external data systems. The README also documents event-time processing, windows, back-pressure, and libraries. The tree and excerpts sample only selected modules, so several core execution stages are represented at module-directory level; arrows show documented workflow or behavior, not uninspected implementation calls.
Main entry points: runtime Flink runtime · CliFrontend.java CLI frontend · PackagedProgram.java Packaged program · executors Deployment executor
Components 20
Submission
- CLI frontend
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java - Packaged program
flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java - Pipeline translator
flink-clients/src/main/java/org/apache/flink/client/FlinkPipelineTranslator.java - Deployment executor
flink-clients/src/main/java/org/apache/flink/client/deployment/executors
Data Processing
- Flink runtime
flink-runtime/src/main/java/org/apache/flink/runtime - Sources
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source - Stream operators
flink-runtime/src/main/java/org/apache/flink/runtime - Sinks
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink - File connector
flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file
Runtime Services
- Checkpoint storage
flink-runtime/src/main/java/org/apache/flink/runtime/state/storage/JobManagerCheckpointStorage.java - State changelog
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskExecutorStateChangelogStoragesManager.java - Leader retrieval
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/LeaderRetrievalService.java - Resource management
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManagerService.java - Tiered shuffle
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageNettyService.java
Monitoring and Access
- Metric query
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java - Web dashboard
flink-runtime-web/web-dashboard/src/app/services/application.service.ts - Queryable state client
flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java
Other components
- Pipeline author
- Flink cluster
- External data systems
Connections 20
- Pipeline author → CLI frontend: submits job
- CLI frontend → Packaged program: loads program
- Packaged program → Pipeline translator: translates pipeline
- CLI frontend → Deployment executor: selects deployment
- Deployment executor → Flink cluster: deploys job
- Flink cluster → Flink runtime: executes job
- Flink runtime → Sources: runs sources
- Flink runtime → Stream operators: runs operators
- Flink runtime → Sinks: runs sinks
- Sources → Stream operators: emits records
- Stream operators → Sinks: forwards records
- File connector → External data systems: reads or writes
- Flink runtime → Checkpoint storage: persists checkpoints
- Flink runtime → State changelog: manages changelog
- Flink runtime → Leader retrieval: retrieves leadership
- Flink runtime → Resource management: manages resources
- Flink runtime → Tiered shuffle: transfers partitions
- Metric query → Web dashboard: serves metrics
- Web dashboard → Flink cluster: requests application data
- Queryable state client → Flink cluster: queries managed state