How apache/seatunnel works
Apache SeaTunnel is a distributed data integration platform: users configure a job and choose an execution engine, then source connectors read data, optional transforms process it, and sink connectors write it. Its connector model supports batch and streaming/CDC synchronization, including multi-table scenarios; the README also highlights distributed snapshots, multimodal data, and real-time monitoring. The API defines connector and table contracts, while the Zeta engine coordinates jobs and exposes runtime services. The graph combines README-level workflow and capabilities with sampled source; some subsystem wiring is documented rather than verified in the excerpts.
Main entry points: connector-starrocks Source Connectors · CoordinatorService.java Zeta Engine · connector-bigquery Sink Connectors · multitablesink Multi-table Sync
Components 21
Connector Contracts
- Source API
seatunnel-api/src/main/java/org/apache/seatunnel/api/source/SeaTunnelSource.java - Sink API
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SeaTunnelSink.java - Table and Schema
seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java
Data Connectors
- Source Connectors
seatunnel-connectors-v2/connector-starrocks - Sink Connectors
seatunnel-connectors-v2/connector-bigquery - Multi-table Sync
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink
Data Processing
- SQL Transform
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngine.java - Multimodal Data
- Distributed Snapshot
Execution Runtime
- Zeta Engine
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java - Flink and Spark
- Engine State Context
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/common/SeaTunnelEngineContext.java
Operations and Monitoring
- Job History
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobHistoryService.java - Realtime Metrics
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/observability/RealtimeMetricsService.java - REST Services
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/BaseService.java - Task Log Cleanup
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogManagerService.java
Other components
- Job Author
- Job Configuration
- Engine Choice
- External Data Sources
- External Data Targets
Connections 24
- Job Author → Job Configuration: defines job
- Job Author → Engine Choice: selects engine
- Job Configuration → Zeta Engine: submits job
- Engine Choice → Zeta Engine: selects runtime
- Engine Choice → Flink and Spark: selects runtime
- Zeta Engine → Source Connectors: runs sources
- Source Connectors → Source API: implements
- Source Connectors → External Data Sources: reads data
- Source Connectors → Table and Schema: uses schemas
- Source Connectors → SQL Transform: provides rows
- SQL Transform → Sink Connectors: transforms rows
- Sink Connectors → Sink API: implements
- Sink Connectors → External Data Targets: writes data
- Sink Connectors → Table and Schema: uses schemas
- Source Connectors → Multi-table Sync: supports sync
- Multi-table Sync → Sink Connectors: coordinates writers
- Source Connectors → Distributed Snapshot: supports consistency
- Source Connectors → Multimodal Data: reads varied formats
- Sink Connectors → Multimodal Data: writes varied formats
- Zeta Engine → Engine State Context: uses state stores
- Zeta Engine → Job History: tracks job state
- Zeta Engine → Realtime Metrics: exposes metrics
- REST Services → Realtime Metrics: serves monitoring
- Zeta Engine → Task Log Cleanup: cleans retained logs