How sujals-codes/olake works
OLake is a database-to-lakehouse replication connector. The principal workflow configures a source and destination, discovers streams, classifies them for full refresh, incremental, or CDC sync, reads records, and writes them to Iceberg or Parquet while tracking state. Source drivers and destination implementations are distinct from the protocol and shared types layers. The README documents supported sources, catalogs, and storage targets; some connector and writer internals were not sampled, so the graph limits detailed runtime links accordingly.
Main entry points: sync.go CLI protocol · interface.go Driver abstraction · OlakeRowsIngester.java Rows ingester · writers.go Writer dispatch
Components 26
Control plane
- Connector entry
connector.go - CLI protocol
protocol/sync.go - Stream discovery
protocol/discover.go
Source connectors
- Driver abstraction
drivers/abstract/interface.go - Postgres driver
drivers/postgres/internal - MySQL driver
drivers/mysql/internal - MongoDB driver
drivers/mongodb/internal - Oracle driver
drivers/oracle/internal - Kafka driver
drivers/kafka/internal - JDBC reader
pkg/jdbc/jdbc.go - Postgres WAL
pkg/waljs/pgoutput.go - Binlog reader
pkg/binlog/binlog.go
Shared sync layer
- Sync modes
drivers/abstract/backfill.go - Sync state
types/state.go - Stream catalog
types/catalog.go - Stream records
types/stream.go
Destination writers
- Iceberg destination
destination/iceberg/iceberg.go - Parquet destination
destination/parquet/parquet.go - Writer dispatch
destination/writers.go
Iceberg writer service
- Java RPC client
destination/iceberg/java_client.go - RPC service
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java - RPC payloads
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngest.java - Rows ingester
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java - Table operations
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/tableoperator/IcebergTableOperator.java - Iceberg catalog
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/IcebergUtil.java
Other components
- Sync operator
Connections 26
- Sync operator → Connector entry: runs
README.md - Connector entry → CLI protocol: registers commands
connector.go - CLI protocol → Stream discovery: discovers streams
- CLI protocol → Driver abstraction: sets up connector
- CLI protocol → Sync modes: classifies streams
protocol/sync.go - CLI protocol → Sync state: loads and updates
- CLI protocol → Stream catalog: reads selection
protocol/sync.go - Postgres driver → Driver abstraction: implements
- MySQL driver → Driver abstraction: implements
- MongoDB driver → Driver abstraction: implements
- Oracle driver → Driver abstraction: implements
- Kafka driver → Driver abstraction: implements
- Postgres driver → Postgres WAL: reads CDC
- MySQL driver → Binlog reader: reads CDC
- Oracle driver → JDBC reader: reads rows
drivers/oracle/internal/backfill.go - Sync modes → Stream records: processes streams
- CLI protocol → Writer dispatch: runs destination
protocol/sync.go - Writer dispatch → Iceberg destination: dispatches writes
destination/writers.go - Writer dispatch → Parquet destination: dispatches writes
destination/writers.go - Iceberg destination → Java RPC client: sends records
- Java RPC client → RPC service: calls RPC
- RPC service → RPC payloads: transports payload
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java - Rows ingester → RPC payloads: handles payload
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java - Rows ingester → Table operations: delegates writes
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java - Rows ingester → Iceberg catalog: uses utilities
destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java - Table operations → Iceberg catalog: operates tables