# datazip-inc/olake-go architecture

> Architecture diagram of the GitHub repository datazip-inc/olake-go, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datazip-inc/olake-go
- Repository: https://github.com/datazip-inc/olake-go

## Overview

OLake Go is a data-replication engine: operators run CLI commands to discover source schemas and sync selected streams through full-load, incremental, or CDC paths into Iceberg or Parquet. Source-specific drivers cover databases, Kafka, and S3; shared driver logic coordinates reads and destination writers. Iceberg includes a Java/gRPC writer service, while Parquet supports filesystem and object-store output. The README documents broader catalog support; sampled excerpts and the source index establish only some internal connections, so unverified links are omitted.

Main entry points: [`drivers/abstract/abstract.go`](https://github.com/datazip-inc/olake-go/blob/master/drivers/abstract/abstract.go) (Driver coordinator), [`destination/writers.go`](https://github.com/datazip-inc/olake-go/blob/master/destination/writers.go) (Writer pool), [`destination/iceberg`](https://github.com/datazip-inc/olake-go/tree/master/destination/iceberg) (Iceberg writer), [`destination/iceberg/java_client.go`](https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/java_client.go) (Iceberg JVM client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Command and orchestration"]
  node_commands["CLI commands<br/>[root.go]"]
  node_driver_api["Driver registration<br/>[connector.go]"]
end

subgraph group_sources["Source connectors"]
  node_db_connectors["Database drivers"]
  node_kafka_driver["Kafka driver"]
  node_s3_driver["S3 driver"]
end

subgraph group_shared["Shared ingestion"]
  node_abstract_driver["Driver coordinator<br/>[abstract.go]"]
  node_jdbc["JDBC access<br/>[jdbc.go]"]
  node_kafka_reader["Kafka reader<br/>[reader.go]"]
  node_s3_parser["S3 parsers"]
  node_stream_types["Stream and state"]
  node_reformat["Record reformatting<br/>[reformat.go]"]
end

subgraph group_destinations["Destination writers"]
  node_writer_pool["Writer pool<br/>[writers.go]"]
  node_iceberg_writer["Iceberg writer"]
  node_parquet_writer["Parquet writer<br/>[parquet.go]"]
end

subgraph group_runtime["Writer runtime"]
  node_indexdb[("Local index store<br/>[pebble.go]")]
  node_java_client["Iceberg JVM client<br/>[java_client.go]"]
  node_record_rpc["Record ingest RPC"]
  node_arrow_rpc["Arrow ingest RPC"]
end

node_operator(("Operator"))
node_iceberg_catalogs[("Iceberg catalogs")]
node_source_systems[("Source systems")]

node_operator -->|"runs"| node_commands
node_commands -->|"orchestrates"| node_abstract_driver
node_driver_api -->|"registers"| node_abstract_driver
node_abstract_driver -->|"creates writers"| node_writer_pool
node_abstract_driver -->|"uses state"| node_stream_types
node_abstract_driver -.->|"delegates source work"| node_db_connectors
node_abstract_driver -.->|"delegates source work"| node_kafka_driver
node_abstract_driver -.->|"delegates source work"| node_s3_driver
node_db_connectors -->|"uses SQL access"| node_jdbc
node_kafka_driver -->|"reads messages"| node_kafka_reader
node_s3_driver -.->|"parses objects"| node_s3_parser
node_source_systems -->|"provides records"| node_db_connectors
node_source_systems -->|"provides messages"| node_kafka_driver
node_source_systems -->|"provides objects"| node_s3_driver
node_abstract_driver -.->|"reformats records"| node_reformat
node_writer_pool -->|"manages indexes"| node_indexdb
node_writer_pool -->|"initializes destination"| node_iceberg_writer
node_writer_pool -->|"initializes destination"| node_parquet_writer
node_iceberg_writer -.->|"starts JVM client"| node_java_client
node_java_client -.->|"calls service"| node_record_rpc
node_java_client -.->|"calls service"| node_arrow_rpc
node_iceberg_writer -->|"writes tables"| node_iceberg_catalogs
node_parquet_writer -->|"writes files"| node_iceberg_catalogs

click node_commands "https://github.com/datazip-inc/olake-go/blob/master/protocol/root.go"
click node_driver_api "https://github.com/datazip-inc/olake-go/blob/master/connector.go"
click node_abstract_driver "https://github.com/datazip-inc/olake-go/blob/master/drivers/abstract/abstract.go"
click node_db_connectors "https://github.com/datazip-inc/olake-go/tree/master/drivers"
click node_kafka_driver "https://github.com/datazip-inc/olake-go/tree/master/drivers/kafka/internal"
click node_s3_driver "https://github.com/datazip-inc/olake-go/tree/master/drivers/s3/internal"
click node_jdbc "https://github.com/datazip-inc/olake-go/blob/master/pkg/jdbc/jdbc.go"
click node_kafka_reader "https://github.com/datazip-inc/olake-go/blob/master/pkg/kafka/reader.go"
click node_s3_parser "https://github.com/datazip-inc/olake-go/tree/master/drivers/s3/internal/pkg/parser"
click node_stream_types "https://github.com/datazip-inc/olake-go/tree/master/types"
click node_reformat "https://github.com/datazip-inc/olake-go/blob/master/utils/typeutils/reformat.go"
click node_writer_pool "https://github.com/datazip-inc/olake-go/blob/master/destination/writers.go"
click node_iceberg_writer "https://github.com/datazip-inc/olake-go/tree/master/destination/iceberg"
click node_parquet_writer "https://github.com/datazip-inc/olake-go/blob/master/destination/parquet/parquet.go"
click node_indexdb "https://github.com/datazip-inc/olake-go/blob/master/pkg/indexdb/pebble.go"
click node_java_client "https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/java_client.go"
click node_record_rpc "https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/RecordIngestServiceGrpc.java"
click node_arrow_rpc "https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/ArrowIngestServiceGrpc.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_commands,node_driver_api toneBlue
class node_db_connectors,node_kafka_driver,node_s3_driver,node_iceberg_catalogs,node_source_systems toneAmber
class node_abstract_driver,node_jdbc,node_kafka_reader,node_s3_parser,node_stream_types,node_reformat toneMint
class node_writer_pool,node_iceberg_writer,node_parquet_writer toneRose
class node_indexdb,node_java_client,node_record_rpc,node_arrow_rpc,node_operator toneIndigo
```

## Components

### Command and orchestration

- [CLI commands](https://github.com/datazip-inc/olake-go/blob/master/protocol/root.go): `protocol/root.go`
- [Driver registration](https://github.com/datazip-inc/olake-go/blob/master/connector.go): `connector.go`

### Source connectors

- [Database drivers](https://github.com/datazip-inc/olake-go/tree/master/drivers): `drivers`
- [Kafka driver](https://github.com/datazip-inc/olake-go/tree/master/drivers/kafka/internal): `drivers/kafka/internal`
- [S3 driver](https://github.com/datazip-inc/olake-go/tree/master/drivers/s3/internal): `drivers/s3/internal`

### Shared ingestion

- [Driver coordinator](https://github.com/datazip-inc/olake-go/blob/master/drivers/abstract/abstract.go): `drivers/abstract/abstract.go`
- [JDBC access](https://github.com/datazip-inc/olake-go/blob/master/pkg/jdbc/jdbc.go): `pkg/jdbc/jdbc.go`
- [Kafka reader](https://github.com/datazip-inc/olake-go/blob/master/pkg/kafka/reader.go): `pkg/kafka/reader.go`
- [S3 parsers](https://github.com/datazip-inc/olake-go/tree/master/drivers/s3/internal/pkg/parser): `drivers/s3/internal/pkg/parser`
- [Stream and state](https://github.com/datazip-inc/olake-go/tree/master/types): `types`
- [Record reformatting](https://github.com/datazip-inc/olake-go/blob/master/utils/typeutils/reformat.go): `utils/typeutils/reformat.go`

### Destination writers

- [Writer pool](https://github.com/datazip-inc/olake-go/blob/master/destination/writers.go): `destination/writers.go`
- [Iceberg writer](https://github.com/datazip-inc/olake-go/tree/master/destination/iceberg): `destination/iceberg`
- [Parquet writer](https://github.com/datazip-inc/olake-go/blob/master/destination/parquet/parquet.go): `destination/parquet/parquet.go`

### Writer runtime

- [Local index store](https://github.com/datazip-inc/olake-go/blob/master/pkg/indexdb/pebble.go): `pkg/indexdb/pebble.go`
- [Iceberg JVM client](https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/java_client.go): `destination/iceberg/java_client.go`
- [Record ingest RPC](https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/RecordIngestServiceGrpc.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/RecordIngestServiceGrpc.java`
- [Arrow ingest RPC](https://github.com/datazip-inc/olake-go/blob/master/destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/ArrowIngestServiceGrpc.java): `destination/iceberg/olake-iceberg-java-writer/src/main/java/io/olake/iceberg/rpc/ArrowIngestServiceGrpc.java`

### Other components

- **Operator**
- **Iceberg catalogs**
- **Source systems**

## Connections

- Operator → CLI commands: runs
- CLI commands → Driver coordinator: orchestrates (evidence: `protocol/root.go`)
- Driver registration → Driver coordinator: registers
- Driver coordinator → Writer pool: creates writers (evidence: `drivers/abstract/abstract.go`)
- Driver coordinator → Stream and state: uses state (evidence: `drivers/abstract/abstract.go`)
- Driver coordinator → Database drivers: delegates source work
- Driver coordinator → Kafka driver: delegates source work
- Driver coordinator → S3 driver: delegates source work
- Database drivers → JDBC access: uses SQL access (evidence: `pkg/jdbc/jdbc.go`)
- Kafka driver → Kafka reader: reads messages (evidence: `drivers/kafka/internal/cdc.go`)
- S3 driver → S3 parsers: parses objects
- Source systems → Database drivers: provides records
- Source systems → Kafka driver: provides messages
- Source systems → S3 driver: provides objects
- Driver coordinator → Record reformatting: reformats records
- Writer pool → Local index store: manages indexes (evidence: `destination/writers.go`)
- Writer pool → Iceberg writer: initializes destination (evidence: `destination/writers.go`)
- Writer pool → Parquet writer: initializes destination (evidence: `destination/writers.go`)
- Iceberg writer → Iceberg JVM client: starts JVM client (evidence: `destination/iceberg/java_client.go`)
- Iceberg JVM client → Record ingest RPC: calls service
- Iceberg JVM client → Arrow ingest RPC: calls service
- Iceberg writer → Iceberg catalogs: writes tables
- Parquet writer → Iceberg catalogs: writes files
