How airbytehq/airbyte works
Airbyte is an open-source data movement platform. The principal workflow is a configured sync: a source connector checks and discovers available streams, reads records and state, then a destination parses and transforms incoming messages, batches them, and writes them to a warehouse, lake, or database. The tree shows substantial bulk source and destination CDK runtime modules, plus connector-specific integrations. The graph focuses on that runtime path; several links between unsampled stages are marked dashed because the excerpts do not establish their dispatch or call sites.
Main entry points: AirbyteConnectorRunner.kt Connector runner · DataFlowPipeline.kt Load pipeline · ReadOperation.kt Source read · CheckOperation.kt Source check
Components 20
Connector runtime
- Connector runner
airbyte-cdk/bulk/core/base/src/main/kotlin/io/airbyte/cdk/AirbyteConnectorRunner.kt - Command configuration
airbyte-cdk/bulk/core/base/src/main/kotlin/io/airbyte/cdk/command/Configuration.kt - Protocol output
airbyte-cdk/bulk/core/base/src/main/kotlin/io/airbyte/cdk/output/StdoutOutputConsumer.kt
Source extraction
- Source check
airbyte-cdk/bulk/core/extract/src/main/kotlin/io/airbyte/cdk/check/CheckOperation.kt - Stream discovery
airbyte-cdk/bulk/core/extract/src/main/kotlin/io/airbyte/cdk/discover/DiscoverOperation.kt - Source read
airbyte-cdk/bulk/core/extract/src/main/kotlin/io/airbyte/cdk/read/ReadOperation.kt - Sync state
airbyte-cdk/bulk/core/extract/src/main/kotlin/io/airbyte/cdk/read/StateManager.kt - Message routing
airbyte-cdk/bulk/core/extract/src/main/kotlin/io/airbyte/cdk/output/OutputMessageRouter.kt
Destination loading
- Destination stream model
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/command/DestinationStream.kt - Message input
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/dataflow/input/DataFlowPipelineInputFlow.kt - Raw record conversion
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/message/DestinationRecordRaw.kt - Load pipeline
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/dataflow/pipeline/DataFlowPipeline.kt - Record aggregation
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/dataflow/aggregate/Aggregate.kt - Destination schema
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/schema/TableSchemaFactory.kt
Destination integrations
- BigQuery file formatting
airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/write/bulk_loader/BigQueryObjectStorageFormattingWriter.kt - BigQuery operations
airbyte-cdk/java/airbyte-cdk/datastore-bigquery/src/main/kotlin/io/airbyte/cdk/db/bigquery/BigQueryDatabase.kt - BigQuery typing
airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/write/typing_deduping/BigQueryDatabaseHandler.kt
Other components
- Sync operator
- Airbyte platform
- Destination store
Connections 20
- Sync operator → Airbyte platform: configures sync
README.md - Airbyte platform → Connector runner: runs connector
- Connector runner → Command configuration: loads configuration
- Connector runner → Source check: dispatches operation
- Connector runner → Stream discovery: dispatches operation
- Connector runner → Source read: dispatches operation
- Source check → Protocol output: emits result
- Stream discovery → Protocol output: emits catalog
- Source read → Sync state: manages state
- Source read → Message routing: routes messages
- Message routing → Airbyte platform: transfers records
- Airbyte platform → Message input: sends sync messages
README.md - Message input → Load pipeline: provides input
- Load pipeline → Raw record conversion: converts records
- Load pipeline → Record aggregation: batches records
- Load pipeline → Destination schema: uses schema
- Record aggregation → BigQuery file formatting: formats batches
- BigQuery file formatting → Destination store: writes files
- BigQuery typing → BigQuery operations: executes SQL
- BigQuery operations → Destination store: queries