How apache/flink-connector-jdbc works
Project type and purpose
• This is an open-source library project: the official Apache Flink JDBC Connector.
• It is structured as a multi-module Maven project that provides:
• A core module with the Flink JDBC connector API and implementation
• A suite of vendor-specific submodules (MySQL, Postgres, Oracle, etc.) that plug into the core via SPI
• Architecture and backward-compatibility test modules
• Documentation and tooling modules
Key modules and their roles
• flink-connector-jdbc-core
• Defines the main runtime components: JdbcSource, JdbcSink, builders, committers, enumerators, readers, dialect interfaces, catalog abstractions, connection providers, transaction managers, and utilities.
• Declares SPI interfaces (org.apache.flink.connector.jdbc.core.database.JdbcFactory) and uses META-INF/services to wire vendor factories.
• flink-connector-jdbc-architecture
• Contains ArchUnit tests enforcing architectural layering and package boundaries on production code.
• flink-connector-jdbc-backward-compatibility
• Contains tests that verify compatibility with previous connector versions across DataStream and Table/SQL APIs.
• Vendor-specific connector modules (flink-connector-jdbc-mysql, ‑postgres, ‑oracle, etc.)
• Each provides an implementation of the JdbcFactory SPI plus dialect, catalog, and type-mapping classes.
• They depend on flink-connector-jdbc-core and register themselves via META-INF/services.
• docs/ and docs/content
• Site content and connector user guides (JDBC DataStream / Table connectors).
• tools/
• CI scripts, Maven checkstyle/release configurations.
Architectural relationships and data flow
• At build time: core → architecture and backward-compat tests run against core (and indirectly vendor modules for compatibility).
• At runtime in Flink:
• Flink job code invokes the public API (JdbcSource/JdbcSink) defined in core.
• Core discovers and loads a vendor’s JdbcFactory via Java’s ServiceLoader and delegates dialect-specific behavior (SQL generation, type mapping, JDBC URL, catalog integration) to that factory.
• Core’s runtime components manage connection providers, split enumeration, record reading/emission, statement execution, and commit/transaction lifecycle.
• Use C4 Component or UML Component view.
• Core (blue)
• Tests (orange)
• Vendor connectors (green)
• Docs and Tools (gray)
• From tests → modules under test
• From vendor connectors → core
• From core → ServiceLoader → vendor connectors
• From Flink runtime (external) → core
• SPI interfaces and service registration (META-INF/services).