How kamisangk/streamcraft works
StreamCraft is a visual Apache Flink pipeline workbench: users author and save a DAG, validate or preview it, submit it to a configured runtime, then monitor and control runs. The service owns APIs, pipeline metadata, and runtime-target management; the core loads definitions and builds Flink jobs from sources, transforms, and sinks. README-documented operator coverage is included, but connector internals beyond the sampled runtime wiring are grouped at their existing source paths. Some links are omitted because their implementation was not sampled.
Main entry points: PipelineApiController.java Pipeline API · PipelineRuntimeQueryService.java Runtime queries · PipelineRuntime.java Pipeline runtime · PipelineExecutionService.java Execution service
Components 26
Authoring
- DAG Studio
service/src/main/resources/static/js/studio-editor.js - Port contracts
service/src/main/resources/static/js/pipeline-port-contract.js
Management service
- Pipeline API
service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java - Definition service
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java - Execution service
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java - Pipeline metadata
service/src/main/java/com/streamcraft/service/pipeline/persistence/PipelineRepository.java - Pipeline record
service/src/main/java/com/streamcraft/service/pipeline/model/Pipeline.java
Flink runtime
- Definition loader
core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java - Pipeline definition
core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java - DAG nodes
core/src/main/java/com/streamcraft/core/model/PipelineNode.java - Runtime validation
core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java - Graph planner
core/src/main/java/com/streamcraft/core/runtime/graph/RuntimeGraphPlanner.java - Pipeline runtime
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java
Connectors
- Source connectors
core/src/main/java/com/streamcraft/core/runtime - Transform operators
core/src/main/java/com/streamcraft/core/runtime/transform/transforms - Sink connectors
core/src/main/java/com/streamcraft/core/runtime - Elasticsearch sink
core/src/main/java/com/streamcraft/core/runtime/ElasticsearchSinkFactory.java - HDFS file sink
core/src/main/java/com/streamcraft/core/runtime/HdfsFileSinkFactory.java
Runtime operations
- Runtime queries
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java - Runtime state
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeStateSupport.java - Pipeline metrics
service/src/main/java/com/streamcraft/service/pipeline/model/PipelineMetrics.java - Target management
service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java - Runtime target
service/src/main/java/com/streamcraft/service/runtime/persistence/FlinkRuntimeTargetRepository.java - Target state
service/src/main/java/com/streamcraft/service/runtime/model/FlinkRuntimeTarget.java
Other components
- Pipeline user
- Flink cluster
Connections 26
- Pipeline user → DAG Studio: authors DAG
README.md - DAG Studio → Port contracts: uses port rules
service/src/main/resources/static/js/pipeline-port-contract.js - Pipeline user → Pipeline API: calls REST API
README.md - Pipeline API → Definition service: gets definitions
service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java - Pipeline API → Execution service: previews or runs
service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java - Pipeline API → Runtime queries: queries status
service/src/main/java/com/streamcraft/service/pipeline/web/PipelineApiController.java - Pipeline API → Pipeline metadata: saves or deletes
- Definition service → Pipeline metadata: reads definitions
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineDefinitionService.java - Execution service → Pipeline metadata: loads run record
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java - Runtime queries → Pipeline metadata: reads pipelines
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java - Runtime queries → Runtime state: resolves runtime state
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java - Runtime queries → Pipeline metrics: returns metrics
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineRuntimeQueryService.java - Target management → Runtime target: loads and saves target
service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java - Target management → Flink cluster: validates connection
service/src/main/java/com/streamcraft/service/runtime/service/FlinkRuntimeTargetService.java - Execution service → Flink cluster: submits or stops jobs
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java - Execution service → Pipeline record: updates run state
service/src/main/java/com/streamcraft/service/pipeline/service/PipelineExecutionService.java - Definition loader → Pipeline definition: loads definition
core/src/main/java/com/streamcraft/core/config/PipelineDefinitionLoader.java - Pipeline definition → DAG nodes: contains DAG nodes
core/src/main/java/com/streamcraft/core/model/PipelineDefinition.java - Runtime validation → Pipeline definition: validates definition
core/src/main/java/com/streamcraft/core/validation/PipelineRuntimeValidator.java - Pipeline runtime → Graph planner: plans graph
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Pipeline runtime → Source connectors: creates sources
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Pipeline runtime → Transform operators: creates operators
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Pipeline runtime → Sink connectors: attaches sinks
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Sink connectors → Elasticsearch sink: includes connector
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Sink connectors → HDFS file sink: includes connector
core/src/main/java/com/streamcraft/core/runtime/PipelineRuntime.java - Flink cluster → Runtime queries: provides job status