How apache/flink-cdc works
Flink CDC is a distributed Flink-based system for real-time and batch change-data capture, synchronization, and transformation. The principal pipeline workflow is to submit a YAML definition, parse source/sink and optional routing and transformation settings, compose a Flink execution, and run it against connectors. The README also documents SQL and DataStream APIs. The sampled sources show definition parsing and Flink composition; connector internals and several runtime subsystems were not sampled, so their implementation-level links are not asserted.
Main entry points: FlinkPipelineComposer.java Flink composer · source Pipeline sources · PipelineDef.java Pipeline definition · flink-cdc-source-connectors SQL connectors
Components 20
Pipeline authoring
- CDC CLI
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java - YAML parser
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java - Pipeline definition
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java - Pipeline options
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java
Composition and runtime
- Flink composer
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java - Pipeline execution
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/PipelineExecution.java - Data transforms
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform - Table routing
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/route - Schema evolution
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema
CDC contracts
- CDC events
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/Event.java - Table schemas
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java
Connector APIs
- Pipeline sources
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source - Pipeline sinks
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink - SQL connectors
flink-cdc-connect/flink-cdc-source-connectors - DataStream sources
flink-cdc-connect/flink-cdc-source-connectors
Other components
- Pipeline user
- External databases
- External destinations
- Flink SQL users
- Flink application
Connections 20
- Pipeline user → CDC CLI: submits YAML
README.md - CDC CLI → YAML parser: parses definition
- YAML parser → Pipeline definition: builds definition
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java - Pipeline definition → Flink composer: composes pipeline
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java - Flink composer → Pipeline execution: creates execution
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java - Flink composer → Pipeline sources: translates source
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java - Flink composer → Data transforms: translates transforms
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java - Flink composer → Table routing: applies routing
- Flink composer → Schema evolution: handles schema changes
- Flink composer → Pipeline sinks: translates sink
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java - Pipeline options → Pipeline definition: configures pipeline
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java - Pipeline sources → External databases: captures changes
- Pipeline sinks → External destinations: writes changes
- Pipeline user → SQL connectors: uses SQL API
README.md - Flink SQL users → SQL connectors: defines CDC source
README.md - SQL connectors → External databases: captures changes
- Pipeline user → DataStream sources: uses DataStream API
README.md - Flink application → DataStream sources: embeds source
README.md - Pipeline sources → CDC events: emits events
- Schema evolution → Table schemas: evolves schemas
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/SchemaManager.java