# apache/flink-cdc architecture

> Architecture diagram of the GitHub repository apache/flink-cdc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/flink-cdc
- Repository: https://github.com/apache/flink-cdc

## Overview

Flink CDC is a distributed Flink-based system for real-time and batch change-data capture, synchronization, and transformation. The principal pipeline workflow is to submit a YAML definition, parse source/sink and optional routing and transformation settings, compose a Flink execution, and run it against connectors. The README also documents SQL and DataStream APIs. The sampled sources show definition parsing and Flink composition; connector internals and several runtime subsystems were not sampled, so their implementation-level links are not asserted.

Main entry points: [`flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`](https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java) (Flink composer), [`flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source`](https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source) (Pipeline sources), [`flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java`](https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java) (Pipeline definition), [`flink-cdc-connect/flink-cdc-source-connectors`](https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-source-connectors) (SQL connectors)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_authoring["Pipeline authoring"]
  node_cli["CDC CLI<br/>[CliFrontend.java]"]
  node_yaml["YAML parser"]
  node_definition["Pipeline definition<br/>[PipelineDef.java]"]
  node_config["Pipeline options"]
end

subgraph group_engine["Composition and runtime"]
  node_composer["Flink composer"]
  node_execution["Pipeline execution"]
  node_transform["Data transforms"]
  node_route["Table routing"]
  node_schemaevo["Schema evolution"]
end

subgraph group_contracts["CDC contracts"]
  node_events["CDC events<br/>[Event.java]"]
  node_schemas["Table schemas<br/>[Schema.java]"]
end

subgraph group_connectors["Connector APIs"]
  node_source["Pipeline sources"]
  node_sink["Pipeline sinks"]
  node_sql["SQL connectors"]
  node_datastream["DataStream sources"]
end

node_user(("Pipeline user"))
node_externaldb[("External databases")]
node_externaldest[("External destinations")]
node_flinksql(("Flink SQL users"))
node_flinkapp(("Flink application"))

node_user -->|"submits YAML"| node_cli
node_cli -.->|"parses definition"| node_yaml
node_yaml -->|"builds definition"| node_definition
node_definition -->|"composes pipeline"| node_composer
node_composer -->|"creates execution"| node_execution
node_composer -->|"translates source"| node_source
node_composer -->|"translates transforms"| node_transform
node_composer -.->|"applies routing"| node_route
node_composer -.->|"handles schema changes"| node_schemaevo
node_composer -->|"translates sink"| node_sink
node_config -->|"configures pipeline"| node_definition
node_source -.->|"captures changes"| node_externaldb
node_sink -.->|"writes changes"| node_externaldest
node_user -->|"uses SQL API"| node_sql
node_flinksql -->|"defines CDC source"| node_sql
node_sql -.->|"captures changes"| node_externaldb
node_user -->|"uses DataStream API"| node_datastream
node_flinkapp -->|"embeds source"| node_datastream
node_source -.->|"emits events"| node_events
node_schemaevo -.->|"evolves schemas"| node_schemas

click node_cli "https://github.com/apache/flink-cdc/blob/master/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java"
click node_yaml "https://github.com/apache/flink-cdc/blob/master/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java"
click node_definition "https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java"
click node_config "https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java"
click node_composer "https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java"
click node_execution "https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/PipelineExecution.java"
click node_source "https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source"
click node_transform "https://github.com/apache/flink-cdc/tree/master/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform"
click node_route "https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/route"
click node_schemaevo "https://github.com/apache/flink-cdc/tree/master/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema"
click node_sink "https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink"
click node_events "https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/Event.java"
click node_schemas "https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java"
click node_sql "https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-source-connectors"
click node_datastream "https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-source-connectors"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_yaml,node_definition,node_config,node_user,node_flinksql toneBlue
class node_composer,node_execution,node_transform,node_route,node_schemaevo,node_externaldb,node_externaldest toneAmber
class node_events,node_schemas toneMint
class node_source,node_sink,node_sql,node_datastream toneRose
class node_flinkapp toneIndigo
```

## Components

### Pipeline authoring

- [CDC CLI](https://github.com/apache/flink-cdc/blob/master/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java): `flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java`
- [YAML parser](https://github.com/apache/flink-cdc/blob/master/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java): `flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java`
- [Pipeline definition](https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java): `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java`
- [Pipeline options](https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java`

### Composition and runtime

- [Flink composer](https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java): `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`
- [Pipeline execution](https://github.com/apache/flink-cdc/blob/master/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/PipelineExecution.java): `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/PipelineExecution.java`
- [Data transforms](https://github.com/apache/flink-cdc/tree/master/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform): `flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform`
- [Table routing](https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/route): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/route`
- [Schema evolution](https://github.com/apache/flink-cdc/tree/master/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema): `flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema`

### CDC contracts

- [CDC events](https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/Event.java): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/Event.java`
- [Table schemas](https://github.com/apache/flink-cdc/blob/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java`

### Connector APIs

- [Pipeline sources](https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source`
- [Pipeline sinks](https://github.com/apache/flink-cdc/tree/master/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink): `flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink`
- [SQL connectors](https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-source-connectors): `flink-cdc-connect/flink-cdc-source-connectors`
- [DataStream sources](https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-source-connectors): `flink-cdc-connect/flink-cdc-source-connectors`

### Other components

- **Pipeline user**
- **External databases**
- **External destinations**
- **Flink SQL users**
- **Flink application**

## Connections

- Pipeline user → CDC CLI: submits YAML (evidence: `README.md`)
- CDC CLI → YAML parser: parses definition
- YAML parser → Pipeline definition: builds definition (evidence: `flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java`)
- Pipeline definition → Flink composer: composes pipeline (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java`)
- Flink composer → Pipeline execution: creates execution (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`)
- Flink composer → Pipeline sources: translates source (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`)
- Flink composer → Data transforms: translates transforms (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`)
- Flink composer → Table routing: applies routing
- Flink composer → Schema evolution: handles schema changes
- Flink composer → Pipeline sinks: translates sink (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java`)
- Pipeline options → Pipeline definition: configures pipeline (evidence: `flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/PipelineDef.java`)
- Pipeline sources → External databases: captures changes
- Pipeline sinks → External destinations: writes changes
- Pipeline user → SQL connectors: uses SQL API (evidence: `README.md`)
- Flink SQL users → SQL connectors: defines CDC source (evidence: `README.md`)
- SQL connectors → External databases: captures changes
- Pipeline user → DataStream sources: uses DataStream API (evidence: `README.md`)
- Flink application → DataStream sources: embeds source (evidence: `README.md`)
- Pipeline sources → CDC events: emits events
- Schema evolution → Table schemas: evolves schemas (evidence: `flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/SchemaManager.java`)