How aritra39514/streamlake works
StreamLake bridges Kafka topics to queryable Iceberg tables. The documented primary workflow registers a table pipeline through the Go API, which persists state and dispatches worker lifecycle operations to the Java data plane; each worker consumes Kafka records and appends Iceberg snapshots. The Java plane also exposes an order producer and optional Kafka Streams analytics. The tree contains a separate Python implementation with API, worker, state, metrics, producer, and writer modules. Relationships are limited to calls shown in excerpts, SOURCE INDEX references, or README documentation; unsampled capabilities are represented without inferred internal wiring.
Main entry points: worker.py Python materializer · Main.java Java entry point · api.py Python API · TableWorker.java Table materializer
Components 22
Control plane
- Control API
control-plane/internal/api/handler.go - SQLite state
control-plane/internal/store/sqlite.go - API and reconciler
control-plane/main.go
Java data plane
- Worker HTTP API
data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java - Worker registry
data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java - Table materializer
data-plane/src/main/java/com/streamlake/worker/TableWorker.java - Order event model
data-plane/src/main/java/com/streamlake/model/Order.java - Iceberg writer
data-plane/src/main/java/com/streamlake/iceberg/IcebergTableWriter.java - Order producer
data-plane/src/main/java/com/streamlake/producer/OrderProducer.java - Java entry point
data-plane/src/main/java/com/streamlake/Main.java
Streaming analytics
- Order metrics streams
data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java - Order serialization
data-plane/src/main/java/com/streamlake/streams/OrderSerde.java
Python runtime
- Python API
streamlake/api.py - Python settings
streamlake/config.py - Python state store
streamlake/database.py - Python materializer
streamlake/worker.py - Python Iceberg writer
streamlake/iceberg_writer.py - Python metrics
streamlake/metrics.py - Python order producer
producer/order_producer.py
External systems
- Kafka topics
- Iceberg tables
Other components
- Pipeline operator
Connections 31
- Pipeline operator → Control API: registers pipeline
README.md - Control API → SQLite state: persists state
control-plane/internal/api/handler.go - API and reconciler → Control API: routes requests
control-plane/main.go - API and reconciler → SQLite state: reconciles state
control-plane/main.go - API and reconciler → Worker HTTP API: dispatches workers
control-plane/main.go - Control API → Worker HTTP API: manages workers
README.md - Worker HTTP API → Worker registry: dispatches lifecycle
data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java - Worker registry → Table materializer: starts and supervises
data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java - Table materializer → Kafka topics: consumes records
data-plane/src/main/java/com/streamlake/worker/TableWorker.java - Table materializer → Order event model: deserializes as
data-plane/src/main/java/com/streamlake/worker/TableWorker.java - Table materializer → Iceberg writer: appends batches
data-plane/src/main/java/com/streamlake/worker/TableWorker.java - Iceberg writer → Iceberg tables: commits snapshots
README.md - Java entry point → Worker HTTP API: starts server
data-plane/src/main/java/com/streamlake/Main.java - Java entry point → Worker registry: creates registry
data-plane/src/main/java/com/streamlake/Main.java - Java entry point → Order producer: runs producer
data-plane/src/main/java/com/streamlake/Main.java - Java entry point → Order metrics streams: starts analytics
data-plane/src/main/java/com/streamlake/Main.java - Order producer → Kafka topics: publishes orders
README.md - Order metrics streams → Kafka topics: reads and writes topics
README.md - Order metrics streams → Order serialization: uses serialization
data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java - Order metrics streams → Order event model: processes orders
data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java - Python API → Python settings: uses settings
streamlake/api.py - Python API → Python state store: reads and writes state
streamlake/api.py - Python API → Python materializer: manages workers
streamlake/api.py - Python API → Python metrics: exposes metrics
streamlake/api.py - Python materializer → Python settings: loads settings
streamlake/worker.py - Python materializer → Python state store: loads and saves offsets
streamlake/worker.py - Python materializer → Python Iceberg writer: writes batches
streamlake/worker.py - Python materializer → Python metrics: updates metrics
streamlake/worker.py - Python materializer → Kafka topics: consumes records
- Python Iceberg writer → Iceberg tables: appends tables
streamlake/iceberg_writer.py - Python order producer → Kafka topics: publishes orders
producer/order_producer.py