# datazip-inc/olake architecture

> Architecture diagram of the GitHub repository datazip-inc/olake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datazip-inc/olake
- Repository: https://github.com/datazip-inc/olake

## Overview

##### Main Components
a. Core Framework
- CLI & HTTP Server (in protocol/)
- State, Config & Logging
- Command handlers: `spec`, `check`, `discover`, `sync`
b. Connectors (drivers/)
- MongoDB connector (CDC & backfill)
- Postgres connector (WAL-based CDC & backfill)
- MySQL connector (Binlog-based CDC & backfill)
- Shared base driver code (drivers/base/)
c. JSON-Schema Generator (jsonschema/)
- Generates CLI spec JSON for UI consumption
d. Data Pipeline Utilities
- Type conversions & flattening (typeutils/, utils/, pkg/binlog, pkg/jdbc, pkg/waljs)
- Catalog & Stream definitions (types/)
- Concurrency & error handling (utils/, safego/)
e. Writers (writers/)
- Apache Iceberg sink (Go + Java/GRPC sub-module)
- Parquet writer
- Local FS & S3 support (via Iceberg and Parquet modules)
f. External Systems
- Source Databases: MongoDB, Postgres, MySQL
- Destination stores: Iceberg Catalog (Glue/Hive/JDBC/REST), S3, Local FS

##### Relationships & Data Flows
- The user invokes OLake CLI or UI →
Core protocol/http_server parses commands →
Core orchestrator loads Connector plugin →
Connector reads from source DB (snapshot + CDC) →
Raw events pass through the data-pipeline utilities (filter, convert, flatten) →
Processed records are handed to Writer plugin →
Writer persists to target (Iceberg, Parquet/S3/FS) →
Core’s HTTP server exposes live metrics (through protocol/http_server)

##### Architectural Patterns
- Plugin-based drivers & writers loaded at runtime via well-defined interfaces (protocol/interface.go, types/adapter.go)
- Command pattern for CLI commands (`spec`, `check`, `discover`, `sync`)
- Factory pattern in JSON-Schema generator for schema attributes
- Concurrency management with safe-go routines (safego/)

##### Important Technologies & Frameworks
- Language: Golang (drivers, core, utilities, writers)
- Embedded Java sub-module for Iceberg sink (Debezium-based gRPC server)
- Protocol: gRPC/HTTP for control plane & metrics
- JSON Schema for CLI/UI integration (rjsf)
- Docker/Docker-Compose for local testing (writers/iceberg/local-test)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External User
    User["User (CLI/UI)"]:::external

    %% Core Framework
    subgraph "Core Framework"
        direction TB
        CLI["CLI"]:::core
        HTTP["HTTP Server"]:::core
        Spec["spec Command"]:::core
        Check["check Command"]:::core
        Discover["discover Command"]:::core
        Sync["sync Command"]:::core
        Interface["Plugin Interface"]:::core
        Adapter["Adapter Interface"]:::core
        Constants["Constants"]:::core
        Logger["Logger"]:::core
    end

    %% Connector Plugins
    subgraph "Connector Plugins"
        direction TB
        BaseDriver["Base Driver"]:::connector
        MongoDB["MongoDB Connector"]:::connector
        Postgres["Postgres Connector"]:::connector
        MySQL["MySQL Connector"]:::connector
    end

    %% Data Pipeline Utilities
    subgraph "Data Pipeline Utilities"
        direction TB
        TypeUtils["Type Conversions"]:::pipeline
        Flatten["Flattening"]:::pipeline
        Binlog["Binlog Utils"]:::pipeline
        JDBC["JDBC Utils"]:::pipeline
        WALJS["WALJS Utils"]:::pipeline
        Catalog["Catalog Definitions"]:::pipeline
        Streams["Stream Definitions"]:::pipeline
        Concurrent["Concurrency Utils"]:::pipeline
        SafeGo["SafeGo"]:::pipeline
    end

    %% Writer Plugins
    subgraph "Writer Plugins"
        direction TB
        Iceberg["Apache Iceberg Sink"]:::writer
        Parquet["Parquet Writer"]:::writer
        LocalS3["Local FS & S3 Support"]:::writer
    end

    %% External Systems
    subgraph "Source Databases & Destinations"
        direction TB
        ExtMongo["MongoDB"]:::external
        ExtPostgres["Postgres"]:::external
        ExtMySQL["MySQL"]:::external
        DestIceberg["Iceberg Catalog"]:::external
        DestS3["S3"]:::external
        DestLocalFS["Local FS"]:::external
    end

    %% Data Flows
    User -->|invokes| CLI
    CLI -->|routes| Spec
    CLI -->|routes| Check
    CLI -->|routes| Discover
    CLI -->|routes| Sync
    CLI -->|uses| HTTP
    Spec --> Interface
    Check --> Interface
    Discover --> Interface
    Sync --> Interface
    Interface --> BaseDriver
    Adapter --> BaseDriver

    %% Connector to Source
    BaseDriver --> MongoDB
    BaseDriver --> Postgres
    BaseDriver --> MySQL
    MongoDB -->|reads| ExtMongo
    Postgres -->|reads| ExtPostgres
    MySQL -->|reads| ExtMySQL

    %% Pipeline flow
    MongoDB --> TypeUtils
    Postgres --> TypeUtils
    MySQL --> TypeUtils
    TypeUtils --> Flatten
    Flatten --> Binlog
    Flatten --> JDBC
    Flatten --> WALJS
    Binlog --> Catalog
    JDBC --> Catalog
    WALJS --> Catalog
    Catalog --> Streams
    Streams --> Concurrent
    Concurrent --> SafeGo

    %% To Writers
    SafeGo --> Iceberg
    SafeGo --> Parquet
    SafeGo --> LocalS3

    %% Writers to Destinations
    Iceberg -->|writes| DestIceberg
    Parquet -->|writes| DestS3
    Parquet -->|writes| DestLocalFS
    Iceberg -->|writes| DestS3
    Iceberg -->|writes| DestLocalFS

    %% Monitoring
    HTTP -->|exposes metrics| User

    %% Click Events for component mapping
    click HTTP "https://github.com/datazip-inc/olake/blob/master/protocol/http_server.go"
    click Spec "https://github.com/datazip-inc/olake/blob/master/protocol/spec.go"
    click Check "https://github.com/datazip-inc/olake/blob/master/protocol/check.go"
    click Discover "https://github.com/datazip-inc/olake/blob/master/protocol/discover.go"
    click Sync "https://github.com/datazip-inc/olake/blob/master/protocol/sync.go"
    click Interface "https://github.com/datazip-inc/olake/blob/master/protocol/interface.go"
    click Adapter "https://github.com/datazip-inc/olake/blob/master/types/adapter.go"
    click Constants "https://github.com/datazip-inc/olake/blob/master/constants/constants.go"
    click Logger "https://github.com/datazip-inc/olake/blob/master/logger/logger.go"
    click BaseDriver "https://github.com/datazip-inc/olake/blob/master/drivers/base/driver.go"
    click BaseDriver "https://github.com/datazip-inc/olake/blob/master/drivers/base/utils.go"
    click MongoDB "https://github.com/datazip-inc/olake/blob/master/drivers/mongodb/main.go"
    click MongoDB "https://github.com/datazip-inc/olake/blob/master/drivers/mongodb/internal/backfill.go"
    click MongoDB "https://github.com/datazip-inc/olake/blob/master/drivers/mongodb/internal/cdc.go"
    click Postgres "https://github.com/datazip-inc/olake/blob/master/drivers/postgres/main.go"
    click Postgres "https://github.com/datazip-inc/olake/blob/master/drivers/postgres/internal/backfill.go"
    click Postgres "https://github.com/datazip-inc/olake/blob/master/drivers/postgres/internal/cdc.go"
    click MySQL "https://github.com/datazip-inc/olake/blob/master/drivers/mysql/main.go"
    click MySQL "https://github.com/datazip-inc/olake/blob/master/drivers/mysql/internal/backfill.go"
    click MySQL "https://github.com/datazip-inc/olake/blob/master/drivers/mysql/internal/cdc.go"
    click Root "https://github.com/datazip-inc/olake/blob/master/jsonschema/cmd/root.go"
    click Gen "https://github.com/datazip-inc/olake/blob/master/jsonschema/generator/generator.go"
    click Schema "https://github.com/datazip-inc/olake/blob/master/jsonschema/schema/jsonschema.go"
    click TypeUtils "https://github.com/datazip-inc/olake/tree/master/typeutils/"
    click Flatten "https://github.com/datazip-inc/olake/blob/master/utils/flatten/flatten.go"
    click Binlog "https://github.com/datazip-inc/olake/blob/master/pkg/binlog/connection.go"
    click JDBC "https://github.com/datazip-inc/olake/blob/master/pkg/jdbc/jdbc.go"
    click WALJS "https://github.com/datazip-inc/olake/blob/master/pkg/waljs/waljs.go"
    click Catalog "https://github.com/datazip-inc/olake/blob/master/types/catalog.go"
    click Streams "https://github.com/datazip-inc/olake/blob/master/types/stream.go"
    click Concurrent "https://github.com/datazip-inc/olake/blob/master/utils/concurrent.go"
    click SafeGo "https://github.com/datazip-inc/olake/blob/master/safego/safego.go"
    click Iceberg "https://github.com/datazip-inc/olake/blob/master/writers/iceberg/iceberg.go"
    click Iceberg "https://github.com/datazip-inc/olake/blob/master/writers/iceberg/iceberg_utils.go"
    click Debezium "https://github.com/datazip-inc/olake/tree/master/writers/iceberg/debezium-server-iceberg-sink/src"
    click Parquet "https://github.com/datazip-inc/olake/blob/master/writers/parquet/parquet.go"
    click ParquetConf "https://github.com/datazip-inc/olake/blob/master/writers/parquet/config.go"
    click IcebergConf "https://github.com/datazip-inc/olake/blob/master/writers/iceberg/config.go"

    %% Styles
    classDef core fill:#D6E9FF,stroke:#333,stroke-width:1px
    classDef connector fill:#DFF0D8,stroke:#333,stroke-width:1px
    classDef pipeline fill:#F5ECCE,stroke:#333,stroke-width:1px
    classDef writer fill:#FDE2D2,stroke:#333,stroke-width:1px
    classDef external fill:#EEEEEE,stroke:#333,stroke-width:1px
```
