# kamisangk/streamcraft architecture

> Architecture diagram of the GitHub repository kamisangk/streamcraft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kamisangk/streamcraft
- Repository: https://github.com/kamisangk/streamcraft

## Overview

StreamCraft is a visual Apache Flink pipeline workbench: users author and save a DAG, validate or preview it, submit it to a configured runtime, then monitor and control runs. The service owns APIs, pipeline metadata, and runtime-target management; the core loads definitions and builds Flink jobs from sources, transforms, and sinks. README-documented operator coverage is included, but connector internals beyond the sampled runtime wiring are grouped at their existing source paths. Some links are omitted because their implementation was not sampled.

Main entry points: [`service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java`](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java) (Pipeline API), [`service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java`](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java) (Runtime queries), [`core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java) (Pipeline runtime), [`service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java`](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java) (Execution service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_author["Authoring"]
  node_studio["DAG Studio<br/>[studio-editor.js]"]
  node_ports["Port contracts"]
end

subgraph group_service["Management service"]
  node_pipeline_api["Pipeline API"]
  node_definition["Definition service"]
  node_execution["Execution service"]
  node_pipeline_store[("Pipeline metadata")]
  node_pipeline_model["Pipeline record<br/>[Pipeline.java]"]
end

subgraph group_runtime["Flink runtime"]
  node_core_loader["Definition loader"]
  node_pipeline_def["Pipeline definition"]
  node_pipeline_node["DAG nodes<br/>[PipelineNode.java]"]
  node_validator["Runtime validation"]
  node_planner["Graph planner"]
  node_pipeline_runtime["Pipeline runtime"]
end

subgraph group_connectors["Connectors"]
  node_sources["Source connectors"]
  node_transforms["Transform operators"]
  node_sinks["Sink connectors"]
  node_es_sink["Elasticsearch sink"]
  node_hdfs_sink["HDFS file sink"]
end

subgraph group_operations["Runtime operations"]
  node_runtime_query["Runtime queries"]
  node_runtime_state["Runtime state"]
  node_metrics["Pipeline metrics"]
  node_target_service["Target management"]
  node_target_store[("Runtime target")]
  node_target_model["Target state"]
end

node_user(("Pipeline user"))
node_flink{{"Flink cluster"}}

node_user -->|"authors DAG"| node_studio
node_studio -->|"uses port rules"| node_ports
node_user -->|"calls REST API"| node_pipeline_api
node_pipeline_api -->|"gets definitions"| node_definition
node_pipeline_api -->|"previews or runs"| node_execution
node_pipeline_api -->|"queries status"| node_runtime_query
node_pipeline_api -.->|"saves or deletes"| node_pipeline_store
node_definition -->|"reads definitions"| node_pipeline_store
node_execution -->|"loads run record"| node_pipeline_store
node_runtime_query -->|"reads pipelines"| node_pipeline_store
node_runtime_query -->|"resolves runtime state"| node_runtime_state
node_runtime_query -->|"returns metrics"| node_metrics
node_target_service -->|"loads and saves target"| node_target_store
node_target_service -->|"validates connection"| node_flink
node_execution -->|"submits or stops jobs"| node_flink
node_execution -->|"updates run state"| node_pipeline_model
node_core_loader -->|"loads definition"| node_pipeline_def
node_pipeline_def -.->|"contains DAG nodes"| node_pipeline_node
node_validator -->|"validates definition"| node_pipeline_def
node_pipeline_runtime -->|"plans graph"| node_planner
node_pipeline_runtime -->|"creates sources"| node_sources
node_pipeline_runtime -->|"creates operators"| node_transforms
node_pipeline_runtime -->|"attaches sinks"| node_sinks
node_sinks -.->|"includes connector"| node_es_sink
node_sinks -.->|"includes connector"| node_hdfs_sink
node_flink -.->|"provides job status"| node_runtime_query

click node_studio "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/resources/static/js/studio-editor.js"
click node_ports "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/resources/static/js/pipeline-port-contract.js"
click node_pipeline_api "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java"
click node_definition "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java"
click node_execution "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java"
click node_pipeline_store "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/persistence/PipelineRepository.java"
click node_pipeline_model "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/model/Pipeline.java"
click node_runtime_query "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java"
click node_runtime_state "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeStateSupport.java"
click node_metrics "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/model/PipelineMetrics.java"
click node_target_service "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java"
click node_target_store "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/persistence/FlinkRuntimeTargetRepository.java"
click node_target_model "https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/model/FlinkRuntimeTarget.java"
click node_core_loader "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java"
click node_pipeline_def "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java"
click node_pipeline_node "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/model/PipelineNode.java"
click node_validator "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java"
click node_planner "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/graph/RuntimeGraphPlanner.java"
click node_pipeline_runtime "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java"
click node_sources "https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime"
click node_transforms "https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime/transform/transforms"
click node_sinks "https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime"
click node_es_sink "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/ElasticsearchSinkFactory.java"
click node_hdfs_sink "https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/HdfsFileSinkFactory.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_studio,node_ports,node_user toneBlue
class node_pipeline_api,node_definition,node_execution,node_pipeline_store,node_pipeline_model toneAmber
class node_core_loader,node_pipeline_def,node_pipeline_node,node_validator,node_planner,node_pipeline_runtime toneMint
class node_sources,node_transforms,node_sinks,node_es_sink,node_hdfs_sink toneRose
class node_runtime_query,node_runtime_state,node_metrics,node_target_service,node_target_store,node_target_model,node_flink toneIndigo
```

## Components

### Authoring

- [DAG Studio](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/resources/static/js/studio-editor.js): `service/src/main/resources/static/js/studio-editor.js`
- [Port contracts](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/resources/static/js/pipeline-port-contract.js): `service/src/main/resources/static/js/pipeline-port-contract.js`

### Management service

- [Pipeline API](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java): `service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java`
- [Definition service](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java): `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java`
- [Execution service](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java): `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java`
- [Pipeline metadata](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/persistence/PipelineRepository.java): `service/src/main/java/com/streamcraft/service/pipeline/persistence/PipelineRepository.java`
- [Pipeline record](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/model/Pipeline.java): `service/src/main/java/com/streamcraft/service/pipeline/model/Pipeline.java`

### Flink runtime

- [Definition loader](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java): `core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java`
- [Pipeline definition](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java): `core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java`
- [DAG nodes](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/model/PipelineNode.java): `core/src/main/java/com/streamcraft/core/model/PipelineNode.java`
- [Runtime validation](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java): `core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java`
- [Graph planner](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/graph/RuntimeGraphPlanner.java): `core/src/main/java/com/streamcraft/core/runtime/graph/RuntimeGraphPlanner.java`
- [Pipeline runtime](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java): `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`

### Connectors

- [Source connectors](https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime): `core/src/main/java/com/streamcraft/core/runtime`
- [Transform operators](https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime/transform/transforms): `core/src/main/java/com/streamcraft/core/runtime/transform/transforms`
- [Sink connectors](https://github.com/kamisangk/streamcraft/tree/main/core/src/main/java/com/streamcraft/core/runtime): `core/src/main/java/com/streamcraft/core/runtime`
- [Elasticsearch sink](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/ElasticsearchSinkFactory.java): `core/src/main/java/com/streamcraft/core/runtime/ElasticsearchSinkFactory.java`
- [HDFS file sink](https://github.com/kamisangk/streamcraft/blob/main/core/src/main/java/com/streamcraft/core/runtime/HdfsFileSinkFactory.java): `core/src/main/java/com/streamcraft/core/runtime/HdfsFileSinkFactory.java`

### Runtime operations

- [Runtime queries](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java): `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java`
- [Runtime state](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeStateSupport.java): `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeStateSupport.java`
- [Pipeline metrics](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/pipeline/model/PipelineMetrics.java): `service/src/main/java/com/streamcraft/service/pipeline/model/PipelineMetrics.java`
- [Target management](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java): `service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java`
- [Runtime target](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/persistence/FlinkRuntimeTargetRepository.java): `service/src/main/java/com/streamcraft/service/runtime/persistence/FlinkRuntimeTargetRepository.java`
- [Target state](https://github.com/kamisangk/streamcraft/blob/main/service/src/main/java/com/streamcraft/service/runtime/model/FlinkRuntimeTarget.java): `service/src/main/java/com/streamcraft/service/runtime/model/FlinkRuntimeTarget.java`

### Other components

- **Pipeline user**
- **Flink cluster**

## Connections

- Pipeline user → DAG Studio: authors DAG (evidence: `README.md`)
- DAG Studio → Port contracts: uses port rules (evidence: `service/src/main/resources/static/js/pipeline-port-contract.js`)
- Pipeline user → Pipeline API: calls REST API (evidence: `README.md`)
- Pipeline API → Definition service: gets definitions (evidence: `service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java`)
- Pipeline API → Execution service: previews or runs (evidence: `service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java`)
- Pipeline API → Runtime queries: queries status (evidence: `service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java`)
- Pipeline API → Pipeline metadata: saves or deletes
- Definition service → Pipeline metadata: reads definitions (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java`)
- Execution service → Pipeline metadata: loads run record (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java`)
- Runtime queries → Pipeline metadata: reads pipelines (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java`)
- Runtime queries → Runtime state: resolves runtime state (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java`)
- Runtime queries → Pipeline metrics: returns metrics (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java`)
- Target management → Runtime target: loads and saves target (evidence: `service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java`)
- Target management → Flink cluster: validates connection (evidence: `service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java`)
- Execution service → Flink cluster: submits or stops jobs (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java`)
- Execution service → Pipeline record: updates run state (evidence: `service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java`)
- Definition loader → Pipeline definition: loads definition (evidence: `core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java`)
- Pipeline definition → DAG nodes: contains DAG nodes (evidence: `core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java`)
- Runtime validation → Pipeline definition: validates definition (evidence: `core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java`)
- Pipeline runtime → Graph planner: plans graph (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Pipeline runtime → Source connectors: creates sources (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Pipeline runtime → Transform operators: creates operators (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Pipeline runtime → Sink connectors: attaches sinks (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Sink connectors → Elasticsearch sink: includes connector (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Sink connectors → HDFS file sink: includes connector (evidence: `core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java`)
- Flink cluster → Runtime queries: provides job status