# aritra39514/streamlake architecture

> Architecture diagram of the GitHub repository aritra39514/streamlake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aritra39514/streamlake
- Repository: https://github.com/aritra39514/streamlake

## Overview

StreamLake bridges Kafka topics to queryable Iceberg tables. The documented primary workflow registers a table pipeline through the Go API, which persists state and dispatches worker lifecycle operations to the Java data plane; each worker consumes Kafka records and appends Iceberg snapshots. The Java plane also exposes an order producer and optional Kafka Streams analytics. The tree contains a separate Python implementation with API, worker, state, metrics, producer, and writer modules. Relationships are limited to calls shown in excerpts, SOURCE INDEX references, or README documentation; unsampled capabilities are represented without inferred internal wiring.

Main entry points: [`streamlake/worker.py`](https://github.com/aritra39514/streamlake/blob/main/streamlake/worker.py) (Python materializer), [`data-plane/src/main/java/com/streamlake/Main.java`](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/Main.java) (Java entry point), [`streamlake/api.py`](https://github.com/aritra39514/streamlake/blob/main/streamlake/api.py) (Python API), [`data-plane/src/main/java/com/streamlake/worker/TableWorker.java`](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/worker/TableWorker.java) (Table materializer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_goapi["Control API<br/>[handler.go]"]
  node_gostore[("SQLite state<br/>[sqlite.go]")]
  node_gomain["API and reconciler<br/>[main.go]"]
end

subgraph group_dataplane["Java data plane"]
  node_jserver["Worker HTTP API"]
  node_registry["Worker registry"]
  node_tableworker["Table materializer<br/>[TableWorker.java]"]
  node_order["Order event model<br/>[Order.java]"]
  node_icewriter["Iceberg writer"]
  node_jproducer["Order producer<br/>[OrderProducer.java]"]
  node_jmain["Java entry point<br/>[Main.java]"]
end

subgraph group_analytics["Streaming analytics"]
  node_streams["Order metrics streams"]
  node_serde["Order serialization<br/>[OrderSerde.java]"]
end

subgraph group_python["Python runtime"]
  node_pythonapi["Python API<br/>[api.py]"]
  node_pyconfig["Python settings<br/>[config.py]"]
  node_pystore[("Python state store<br/>[database.py]")]
  node_pyworker["Python materializer<br/>[worker.py]"]
  node_pywriter["Python Iceberg writer<br/>[iceberg_writer.py]"]
  node_pymetrics["Python metrics<br/>[metrics.py]"]
  node_pyproducer["Python order producer<br/>[order_producer.py]"]
end

subgraph group_services["External systems"]
  node_kafka["Kafka topics"]
  node_iceberg[("Iceberg tables")]
end

node_operator(("Pipeline operator"))

node_operator -->|"registers pipeline"| node_goapi
node_goapi -->|"persists state"| node_gostore
node_gomain -->|"routes requests"| node_goapi
node_gomain -->|"reconciles state"| node_gostore
node_gomain -->|"dispatches workers"| node_jserver
node_goapi -->|"manages workers"| node_jserver
node_jserver -->|"dispatches lifecycle"| node_registry
node_registry -->|"starts and supervises"| node_tableworker
node_tableworker -->|"consumes records"| node_kafka
node_tableworker -->|"deserializes as"| node_order
node_tableworker -->|"appends batches"| node_icewriter
node_icewriter -->|"commits snapshots"| node_iceberg
node_jmain -->|"starts server"| node_jserver
node_jmain -->|"creates registry"| node_registry
node_jmain -->|"runs producer"| node_jproducer
node_jmain -->|"starts analytics"| node_streams
node_jproducer -->|"publishes orders"| node_kafka
node_streams -->|"reads and writes topics"| node_kafka
node_streams -->|"uses serialization"| node_serde
node_streams -->|"processes orders"| node_order
node_pythonapi -->|"uses settings"| node_pyconfig
node_pythonapi -->|"reads and writes state"| node_pystore
node_pythonapi -->|"manages workers"| node_pyworker
node_pythonapi -->|"exposes metrics"| node_pymetrics
node_pyworker -->|"loads settings"| node_pyconfig
node_pyworker -->|"loads and saves offsets"| node_pystore
node_pyworker -->|"writes batches"| node_pywriter
node_pyworker -->|"updates metrics"| node_pymetrics
node_pyworker -.->|"consumes records"| node_kafka
node_pywriter -->|"appends tables"| node_iceberg
node_pyproducer -->|"publishes orders"| node_kafka

click node_goapi "https://github.com/aritra39514/streamlake/blob/main/control-plane/internal/api/handler.go"
click node_gostore "https://github.com/aritra39514/streamlake/blob/main/control-plane/internal/store/sqlite.go"
click node_gomain "https://github.com/aritra39514/streamlake/blob/main/control-plane/main.go"
click node_jserver "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java"
click node_registry "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java"
click node_tableworker "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/worker/TableWorker.java"
click node_order "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/model/Order.java"
click node_icewriter "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/iceberg/IcebergTableWriter.java"
click node_jproducer "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/producer/OrderProducer.java"
click node_jmain "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/Main.java"
click node_streams "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java"
click node_serde "https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/streams/OrderSerde.java"
click node_pythonapi "https://github.com/aritra39514/streamlake/blob/main/streamlake/api.py"
click node_pyconfig "https://github.com/aritra39514/streamlake/blob/main/streamlake/config.py"
click node_pystore "https://github.com/aritra39514/streamlake/blob/main/streamlake/database.py"
click node_pyworker "https://github.com/aritra39514/streamlake/blob/main/streamlake/worker.py"
click node_pywriter "https://github.com/aritra39514/streamlake/blob/main/streamlake/iceberg_writer.py"
click node_pymetrics "https://github.com/aritra39514/streamlake/blob/main/streamlake/metrics.py"
click node_pyproducer "https://github.com/aritra39514/streamlake/blob/main/producer/order_producer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_goapi,node_gostore,node_gomain toneBlue
class node_jserver,node_registry,node_tableworker,node_order,node_icewriter,node_jproducer,node_jmain toneAmber
class node_streams,node_serde toneMint
class node_pythonapi,node_pyconfig,node_pystore,node_pyworker,node_pywriter,node_pymetrics,node_pyproducer toneRose
class node_kafka,node_iceberg,node_operator toneIndigo
```

## Components

### Control plane

- [Control API](https://github.com/aritra39514/streamlake/blob/main/control-plane/internal/api/handler.go): `control-plane/internal/api/handler.go`
- [SQLite state](https://github.com/aritra39514/streamlake/blob/main/control-plane/internal/store/sqlite.go): `control-plane/internal/store/sqlite.go`
- [API and reconciler](https://github.com/aritra39514/streamlake/blob/main/control-plane/main.go): `control-plane/main.go`

### Java data plane

- [Worker HTTP API](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java): `data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java`
- [Worker registry](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java): `data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java`
- [Table materializer](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/worker/TableWorker.java): `data-plane/src/main/java/com/streamlake/worker/TableWorker.java`
- [Order event model](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/model/Order.java): `data-plane/src/main/java/com/streamlake/model/Order.java`
- [Iceberg writer](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/iceberg/IcebergTableWriter.java): `data-plane/src/main/java/com/streamlake/iceberg/IcebergTableWriter.java`
- [Order producer](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/producer/OrderProducer.java): `data-plane/src/main/java/com/streamlake/producer/OrderProducer.java`
- [Java entry point](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/Main.java): `data-plane/src/main/java/com/streamlake/Main.java`

### Streaming analytics

- [Order metrics streams](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java): `data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java`
- [Order serialization](https://github.com/aritra39514/streamlake/blob/main/data-plane/src/main/java/com/streamlake/streams/OrderSerde.java): `data-plane/src/main/java/com/streamlake/streams/OrderSerde.java`

### Python runtime

- [Python API](https://github.com/aritra39514/streamlake/blob/main/streamlake/api.py): `streamlake/api.py`
- [Python settings](https://github.com/aritra39514/streamlake/blob/main/streamlake/config.py): `streamlake/config.py`
- [Python state store](https://github.com/aritra39514/streamlake/blob/main/streamlake/database.py): `streamlake/database.py`
- [Python materializer](https://github.com/aritra39514/streamlake/blob/main/streamlake/worker.py): `streamlake/worker.py`
- [Python Iceberg writer](https://github.com/aritra39514/streamlake/blob/main/streamlake/iceberg_writer.py): `streamlake/iceberg_writer.py`
- [Python metrics](https://github.com/aritra39514/streamlake/blob/main/streamlake/metrics.py): `streamlake/metrics.py`
- [Python order producer](https://github.com/aritra39514/streamlake/blob/main/producer/order_producer.py): `producer/order_producer.py`

### External systems

- **Kafka topics**
- **Iceberg tables**

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Control API: registers pipeline (evidence: `README.md`)
- Control API → SQLite state: persists state (evidence: `control-plane/internal/api/handler.go`)
- API and reconciler → Control API: routes requests (evidence: `control-plane/main.go`)
- API and reconciler → SQLite state: reconciles state (evidence: `control-plane/main.go`)
- API and reconciler → Worker HTTP API: dispatches workers (evidence: `control-plane/main.go`)
- Control API → Worker HTTP API: manages workers (evidence: `README.md`)
- Worker HTTP API → Worker registry: dispatches lifecycle (evidence: `data-plane/src/main/java/com/streamlake/server/DataPlaneServer.java`)
- Worker registry → Table materializer: starts and supervises (evidence: `data-plane/src/main/java/com/streamlake/worker/WorkerRegistry.java`)
- Table materializer → Kafka topics: consumes records (evidence: `data-plane/src/main/java/com/streamlake/worker/TableWorker.java`)
- Table materializer → Order event model: deserializes as (evidence: `data-plane/src/main/java/com/streamlake/worker/TableWorker.java`)
- Table materializer → Iceberg writer: appends batches (evidence: `data-plane/src/main/java/com/streamlake/worker/TableWorker.java`)
- Iceberg writer → Iceberg tables: commits snapshots (evidence: `README.md`)
- Java entry point → Worker HTTP API: starts server (evidence: `data-plane/src/main/java/com/streamlake/Main.java`)
- Java entry point → Worker registry: creates registry (evidence: `data-plane/src/main/java/com/streamlake/Main.java`)
- Java entry point → Order producer: runs producer (evidence: `data-plane/src/main/java/com/streamlake/Main.java`)
- Java entry point → Order metrics streams: starts analytics (evidence: `data-plane/src/main/java/com/streamlake/Main.java`)
- Order producer → Kafka topics: publishes orders (evidence: `README.md`)
- Order metrics streams → Kafka topics: reads and writes topics (evidence: `README.md`)
- Order metrics streams → Order serialization: uses serialization (evidence: `data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java`)
- Order metrics streams → Order event model: processes orders (evidence: `data-plane/src/main/java/com/streamlake/streams/OrderMetricsStream.java`)
- Python API → Python settings: uses settings (evidence: `streamlake/api.py`)
- Python API → Python state store: reads and writes state (evidence: `streamlake/api.py`)
- Python API → Python materializer: manages workers (evidence: `streamlake/api.py`)
- Python API → Python metrics: exposes metrics (evidence: `streamlake/api.py`)
- Python materializer → Python settings: loads settings (evidence: `streamlake/worker.py`)
- Python materializer → Python state store: loads and saves offsets (evidence: `streamlake/worker.py`)
- Python materializer → Python Iceberg writer: writes batches (evidence: `streamlake/worker.py`)
- Python materializer → Python metrics: updates metrics (evidence: `streamlake/worker.py`)
- Python materializer → Kafka topics: consumes records
- Python Iceberg writer → Iceberg tables: appends tables (evidence: `streamlake/iceberg_writer.py`)
- Python order producer → Kafka topics: publishes orders (evidence: `producer/order_producer.py`)
