# sujals-codes/olake architecture

> Architecture diagram of the GitHub repository sujals-codes/olake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sujals-codes/olake
- Repository: https://github.com/sujals-codes/olake

## Overview

OLake is a database-to-lakehouse replication connector. The principal workflow configures a source and destination, discovers streams, classifies them for full refresh, incremental, or CDC sync, reads records, and writes them to Iceberg or Parquet while tracking state. Source drivers and destination implementations are distinct from the protocol and shared types layers. The README documents supported sources, catalogs, and storage targets; some connector and writer internals were not sampled, so the graph limits detailed runtime links accordingly.

Main entry points: [`protocol/sync.go`](https://github.com/sujals-codes/olake/blob/master/protocol/sync.go) (CLI protocol), [`drivers/abstract/interface.go`](https://github.com/sujals-codes/olake/blob/master/drivers/abstract/interface.go) (Driver abstraction), [`destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java`](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java) (Rows ingester), [`destination/writers.go`](https://github.com/sujals-codes/olake/blob/master/destination/writers.go) (Writer dispatch)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_connector["Connector entry<br/>[connector.go]"]
  node_protocol["CLI protocol<br/>[sync.go]"]
  node_discovery["Stream discovery<br/>[discover.go]"]
end

subgraph group_sources["Source connectors"]
  node_driverapi["Driver abstraction<br/>[interface.go]"]
  node_postgres["Postgres driver"]
  node_mysql["MySQL driver"]
  node_mongodb["MongoDB driver"]
  node_oracle["Oracle driver"]
  node_kafka["Kafka driver"]
  node_jdbc["JDBC reader<br/>[jdbc.go]"]
  node_pgwal["Postgres WAL<br/>[pgoutput.go]"]
  node_binlog["Binlog reader<br/>[binlog.go]"]
end

subgraph group_shared["Shared sync layer"]
  node_syncmodes["Sync modes<br/>[backfill.go]"]
  node_state[("Sync state<br/>[state.go]")]
  node_catalog["Stream catalog<br/>[catalog.go]"]
  node_types["Stream records<br/>[stream.go]"]
end

subgraph group_outputs["Destination writers"]
  node_icebergwriter["Iceberg destination<br/>[iceberg.go]"]
  node_parquet["Parquet destination<br/>[parquet.go]"]
  node_writerapi["Writer dispatch<br/>[writers.go]"]
end

subgraph group_iceberg["Iceberg writer service"]
  node_javaclient["Java RPC client<br/>[java_client.go]"]
  node_grpcservice["RPC service"]
  node_rpcpayload["RPC payloads<br/>[RecordIngest.java]"]
  node_ingester["Rows ingester"]
  node_tableoperator["Table operations"]
  node_icebergcatalog[("Iceberg catalog<br/>[IcebergUtil.java]")]
end

node_operator(("Sync operator"))

node_operator -->|"runs"| node_connector
node_connector -->|"registers commands"| node_protocol
node_protocol -.->|"discovers streams"| node_discovery
node_protocol -->|"sets up connector"| node_driverapi
node_protocol -->|"classifies streams"| node_syncmodes
node_protocol -.->|"loads and updates"| node_state
node_protocol -->|"reads selection"| node_catalog
node_postgres -.->|"implements"| node_driverapi
node_mysql -.->|"implements"| node_driverapi
node_mongodb -.->|"implements"| node_driverapi
node_oracle -.->|"implements"| node_driverapi
node_kafka -.->|"implements"| node_driverapi
node_postgres -.->|"reads CDC"| node_pgwal
node_mysql -.->|"reads CDC"| node_binlog
node_oracle -->|"reads rows"| node_jdbc
node_syncmodes -.->|"processes streams"| node_types
node_protocol -->|"runs destination"| node_writerapi
node_writerapi -->|"dispatches writes"| node_icebergwriter
node_writerapi -->|"dispatches writes"| node_parquet
node_icebergwriter -.->|"sends records"| node_javaclient
node_javaclient -.->|"calls RPC"| node_grpcservice
node_grpcservice -->|"transports payload"| node_rpcpayload
node_ingester -.->|"handles payload"| node_rpcpayload
node_ingester -.->|"delegates writes"| node_tableoperator
node_ingester -.->|"uses utilities"| node_icebergcatalog
node_tableoperator -.->|"operates tables"| node_icebergcatalog

click node_connector "https://github.com/sujals-codes/olake/blob/master/connector.go"
click node_protocol "https://github.com/sujals-codes/olake/blob/master/protocol/sync.go"
click node_discovery "https://github.com/sujals-codes/olake/blob/master/protocol/discover.go"
click node_driverapi "https://github.com/sujals-codes/olake/blob/master/drivers/abstract/interface.go"
click node_postgres "https://github.com/sujals-codes/olake/tree/master/drivers/postgres/internal"
click node_mysql "https://github.com/sujals-codes/olake/tree/master/drivers/mysql/internal"
click node_mongodb "https://github.com/sujals-codes/olake/tree/master/drivers/mongodb/internal"
click node_oracle "https://github.com/sujals-codes/olake/tree/master/drivers/oracle/internal"
click node_kafka "https://github.com/sujals-codes/olake/tree/master/drivers/kafka/internal"
click node_jdbc "https://github.com/sujals-codes/olake/blob/master/pkg/jdbc/jdbc.go"
click node_pgwal "https://github.com/sujals-codes/olake/blob/master/pkg/waljs/pgoutput.go"
click node_binlog "https://github.com/sujals-codes/olake/blob/master/pkg/binlog/binlog.go"
click node_syncmodes "https://github.com/sujals-codes/olake/blob/master/drivers/abstract/backfill.go"
click node_state "https://github.com/sujals-codes/olake/blob/master/types/state.go"
click node_catalog "https://github.com/sujals-codes/olake/blob/master/types/catalog.go"
click node_types "https://github.com/sujals-codes/olake/blob/master/types/stream.go"
click node_icebergwriter "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/iceberg.go"
click node_parquet "https://github.com/sujals-codes/olake/blob/master/destination/parquet/parquet.go"
click node_writerapi "https://github.com/sujals-codes/olake/blob/master/destination/writers.go"
click node_javaclient "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/java_client.go"
click node_grpcservice "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java"
click node_rpcpayload "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngest.java"
click node_ingester "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java"
click node_tableoperator "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/tableoperator/IcebergTableOperator.java"
click node_icebergcatalog "https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/IcebergUtil.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_connector,node_protocol,node_discovery toneBlue
class node_driverapi,node_postgres,node_mysql,node_mongodb,node_oracle,node_kafka,node_jdbc,node_pgwal,node_binlog toneAmber
class node_syncmodes,node_state,node_catalog,node_types toneMint
class node_icebergwriter,node_parquet,node_writerapi toneRose
class node_javaclient,node_grpcservice,node_rpcpayload,node_ingester,node_tableoperator,node_icebergcatalog,node_operator toneIndigo
```

## Components

### Control plane

- [Connector entry](https://github.com/sujals-codes/olake/blob/master/connector.go): `connector.go`
- [CLI protocol](https://github.com/sujals-codes/olake/blob/master/protocol/sync.go): `protocol/sync.go`
- [Stream discovery](https://github.com/sujals-codes/olake/blob/master/protocol/discover.go): `protocol/discover.go`

### Source connectors

- [Driver abstraction](https://github.com/sujals-codes/olake/blob/master/drivers/abstract/interface.go): `drivers/abstract/interface.go`
- [Postgres driver](https://github.com/sujals-codes/olake/tree/master/drivers/postgres/internal): `drivers/postgres/internal`
- [MySQL driver](https://github.com/sujals-codes/olake/tree/master/drivers/mysql/internal): `drivers/mysql/internal`
- [MongoDB driver](https://github.com/sujals-codes/olake/tree/master/drivers/mongodb/internal): `drivers/mongodb/internal`
- [Oracle driver](https://github.com/sujals-codes/olake/tree/master/drivers/oracle/internal): `drivers/oracle/internal`
- [Kafka driver](https://github.com/sujals-codes/olake/tree/master/drivers/kafka/internal): `drivers/kafka/internal`
- [JDBC reader](https://github.com/sujals-codes/olake/blob/master/pkg/jdbc/jdbc.go): `pkg/jdbc/jdbc.go`
- [Postgres WAL](https://github.com/sujals-codes/olake/blob/master/pkg/waljs/pgoutput.go): `pkg/waljs/pgoutput.go`
- [Binlog reader](https://github.com/sujals-codes/olake/blob/master/pkg/binlog/binlog.go): `pkg/binlog/binlog.go`

### Shared sync layer

- [Sync modes](https://github.com/sujals-codes/olake/blob/master/drivers/abstract/backfill.go): `drivers/abstract/backfill.go`
- [Sync state](https://github.com/sujals-codes/olake/blob/master/types/state.go): `types/state.go`
- [Stream catalog](https://github.com/sujals-codes/olake/blob/master/types/catalog.go): `types/catalog.go`
- [Stream records](https://github.com/sujals-codes/olake/blob/master/types/stream.go): `types/stream.go`

### Destination writers

- [Iceberg destination](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/iceberg.go): `destination/iceberg/iceberg.go`
- [Parquet destination](https://github.com/sujals-codes/olake/blob/master/destination/parquet/parquet.go): `destination/parquet/parquet.go`
- [Writer dispatch](https://github.com/sujals-codes/olake/blob/master/destination/writers.go): `destination/writers.go`

### Iceberg writer service

- [Java RPC client](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/java_client.go): `destination/iceberg/java_client.go`
- [RPC service](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java`
- [RPC payloads](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngest.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngest.java`
- [Rows ingester](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java`
- [Table operations](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/tableoperator/IcebergTableOperator.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/tableoperator/IcebergTableOperator.java`
- [Iceberg catalog](https://github.com/sujals-codes/olake/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/IcebergUtil.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/IcebergUtil.java`

### Other components

- **Sync operator**

## Connections

- Sync operator → Connector entry: runs (evidence: `README.md`)
- Connector entry → CLI protocol: registers commands (evidence: `connector.go`)
- CLI protocol → Stream discovery: discovers streams
- CLI protocol → Driver abstraction: sets up connector
- CLI protocol → Sync modes: classifies streams (evidence: `protocol/sync.go`)
- CLI protocol → Sync state: loads and updates
- CLI protocol → Stream catalog: reads selection (evidence: `protocol/sync.go`)
- Postgres driver → Driver abstraction: implements
- MySQL driver → Driver abstraction: implements
- MongoDB driver → Driver abstraction: implements
- Oracle driver → Driver abstraction: implements
- Kafka driver → Driver abstraction: implements
- Postgres driver → Postgres WAL: reads CDC
- MySQL driver → Binlog reader: reads CDC
- Oracle driver → JDBC reader: reads rows (evidence: `drivers/oracle/internal/backfill.go`)
- Sync modes → Stream records: processes streams
- CLI protocol → Writer dispatch: runs destination (evidence: `protocol/sync.go`)
- Writer dispatch → Iceberg destination: dispatches writes (evidence: `destination/writers.go`)
- Writer dispatch → Parquet destination: dispatches writes (evidence: `destination/writers.go`)
- Iceberg destination → Java RPC client: sends records
- Java RPC client → RPC service: calls RPC
- RPC service → RPC payloads: transports payload (evidence: `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/RecordIngestServiceGrpc.java`)
- Rows ingester → RPC payloads: handles payload (evidence: `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java`)
- Rows ingester → Table operations: delegates writes (evidence: `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java`)
- Rows ingester → Iceberg catalog: uses utilities (evidence: `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/debezium/server/iceberg/rpc/OlakeRowsIngester.java`)
- Table operations → Iceberg catalog: operates tables
