How datazip-inc/olake works
Main Components
a. Core Framework
• CLI & HTTP Server (in protocol/)
• State, Config & Logging
• Command handlers: spec, check, discover, sync
b. Connectors (drivers/)
• MongoDB connector (CDC & backfill)
• Postgres connector (WAL-based CDC & backfill)
• MySQL connector (Binlog-based CDC & backfill)
• Shared base driver code (drivers/base/)
c. JSON-Schema Generator (jsonschema/)
• Generates CLI spec JSON for UI consumption
d. Data Pipeline Utilities
• Type conversions & flattening (typeutils/, utils/, pkg/binlog, pkg/jdbc, pkg/waljs)
• Catalog & Stream definitions (types/)
• Concurrency & error handling (utils/, safego/)
e. Writers (writers/)
• Apache Iceberg sink (Go + Java/GRPC sub-module)
• Parquet writer
• Local FS & S3 support (via Iceberg and Parquet modules)
f. External Systems
• Source Databases: MongoDB, Postgres, MySQL
• Destination stores: Iceberg Catalog (Glue/Hive/JDBC/REST), S3, Local FS
Relationships & Data Flows
• The user invokes OLake CLI or UI →
Core protocol/http_server parses commands →
Core orchestrator loads Connector plugin →
Connector reads from source DB (snapshot + CDC) →
Raw events pass through the data-pipeline utilities (filter, convert, flatten) →
Processed records are handed to Writer plugin →
Writer persists to target (Iceberg, Parquet/S3/FS) →
Core’s HTTP server exposes live metrics (through protocol/http_server)
Architectural Patterns
• Plugin-based drivers & writers loaded at runtime via well-defined interfaces (protocol/interface.go, types/adapter.go)
• Command pattern for CLI commands (spec, check, discover, sync)
• Factory pattern in JSON-Schema generator for schema attributes
• Concurrency management with safe-go routines (safego/)
Important Technologies & Frameworks
• Language: Golang (drivers, core, utilities, writers)
• Embedded Java sub-module for Iceberg sink (Debezium-based gRPC server)
• Protocol: gRPC/HTTP for control plane & metrics
• JSON Schema for CLI/UI integration (rjsf)
• Docker/Docker-Compose for local testing (writers/iceberg/local-test)