How datazip-inc/olake-go works
OLake Go is a data-replication engine: operators run CLI commands to discover source schemas and sync selected streams through full-load, incremental, or CDC paths into Iceberg or Parquet. Source-specific drivers cover databases, Kafka, and S3; shared driver logic coordinates reads and destination writers. Iceberg includes a Java/gRPC writer service, while Parquet supports filesystem and object-store output. The README documents broader catalog support; sampled excerpts and the source index establish only some internal connections, so unverified links are omitted.
Main entry points: abstract.go Driver coordinator · writers.go Writer pool · iceberg Iceberg writer · java_client.go Iceberg JVM client
Components 21
Command and orchestration
- CLI commands
protocol/root.go - Driver registration
connector.go
Source connectors
- Database drivers
drivers - Kafka driver
drivers/kafka/internal - S3 driver
drivers/s3/internal
Shared ingestion
- Driver coordinator
drivers/abstract/abstract.go - JDBC access
pkg/jdbc/jdbc.go - Kafka reader
pkg/kafka/reader.go - S3 parsers
drivers/s3/internal/pkg/parser - Stream and state
types - Record reformatting
utils/typeutils/reformat.go
Destination writers
- Writer pool
destination/writers.go - Iceberg writer
destination/iceberg - Parquet writer
destination/parquet/parquet.go
Writer runtime
- Local index store
pkg/indexdb/pebble.go - Iceberg JVM client
destination/iceberg/java_client.go - Record ingest RPC
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/RecordIngestServiceGrpc.java - Arrow ingest RPC
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/ArrowIngestServiceGrpc.java
Other components
- Operator
- Iceberg catalogs
- Source systems
Connections 23
- Operator → CLI commands: runs
- CLI commands → Driver coordinator: orchestrates
protocol/root.go - Driver registration → Driver coordinator: registers
- Driver coordinator → Writer pool: creates writers
drivers/abstract/abstract.go - Driver coordinator → Stream and state: uses state
drivers/abstract/abstract.go - Driver coordinator → Database drivers: delegates source work
- Driver coordinator → Kafka driver: delegates source work
- Driver coordinator → S3 driver: delegates source work
- Database drivers → JDBC access: uses SQL access
pkg/jdbc/jdbc.go - Kafka driver → Kafka reader: reads messages
drivers/kafka/internal/cdc.go - S3 driver → S3 parsers: parses objects
- Source systems → Database drivers: provides records
- Source systems → Kafka driver: provides messages
- Source systems → S3 driver: provides objects
- Driver coordinator → Record reformatting: reformats records
- Writer pool → Local index store: manages indexes
destination/writers.go - Writer pool → Iceberg writer: initializes destination
destination/writers.go - Writer pool → Parquet writer: initializes destination
destination/writers.go - Iceberg writer → Iceberg JVM client: starts JVM client
destination/iceberg/java_client.go - Iceberg JVM client → Record ingest RPC: calls service
- Iceberg JVM client → Arrow ingest RPC: calls service
- Iceberg writer → Iceberg catalogs: writes tables
- Parquet writer → Iceberg catalogs: writes files