How tarungka/wire works
Wire is a distributed stream-processing engine: users define a job through the Go SDK or submit/manage jobs through the CLI and coordinator API; the coordinator schedules work to workers, which execute stream tasks and emit results. Checkpointing and persistent state support recovery, while RPC and transport connect the cluster. The README also identifies exactly-once processing, event ordering, HA, and observability as product capabilities. The excerpts show coordinator dispatch and worker execution; links for unsampled subsystems are limited to relationships indicated by the documented runtime and source names.
Main entry points: worker.go Worker Service · main.go Binary Entrypoint · environment.go Go SDK · job_manager.go Job Management
Components 22
Job Authoring
- Go SDK
sdk/environment.go - YAML Pipelines
sdk/pipeline_yaml.go - Connector Adapters
sdk/connectors
Control Plane
- CLI Commands
internal/jobcli/run.go - Binary Entrypoint
cmd/main.go - Coordinator API
internal/coordinator/http.go - Job Management
internal/coordinator/job_manager.go - Task Scheduling
internal/coordinator/scheduler.go - Leader Election
internal/coordinator/election.go
Execution Runtime
- Worker Service
internal/worker/worker.go - Task Engine
internal/engine/operator_chain.go - Stream I/O
internal/engine/input_reader.go
State and Recovery
- Coordinator Store
internal/coordinator/store.go - Checkpoint Control
internal/coordinator/checkpoint_manager.go - Task State
internal/engine/state_backend.go - Pebble State
internal/engine/state_backend_pebble.go
Connectivity and Operations
- RPC Services
internal/rpc/server.go - Data Transport
internal/transport/data_stream.go - Metrics and Telemetry
internal/observability/otel.go
Other components
- Job Author
- External Data Systems
- Kubernetes Leases
Connections 27
- Job Author → Go SDK: defines jobs
- Go SDK → YAML Pipelines: parses pipeline
- Go SDK → Connector Adapters: resolves connectors
- Job Author → CLI Commands: runs commands
- CLI Commands → Binary Entrypoint: invokes binary
- Binary Entrypoint → Worker Service: starts worker
- Binary Entrypoint → Coordinator API: starts coordinator
- Job Author → Coordinator API: submits jobs
- Coordinator API → Job Management: manages jobs
- Job Management → Coordinator Store: persists metadata
- Job Management → Task Scheduling: wakes scheduler
- Task Scheduling → RPC Services: dispatches tasks
- RPC Services → Worker Service: delivers commands
- Worker Service → Task Engine: executes tasks
- Task Engine → Stream I/O: reads and writes
- Stream I/O → External Data Systems: exchanges events
- Worker Service → Data Transport: uses data streams
- Data Transport → Worker Service: delivers task data
- Task Engine → Task State: uses keyed state
- Task State → Pebble State: persists state
- Checkpoint Control → RPC Services: triggers snapshots
- Checkpoint Control → Coordinator Store: stores checkpoint records
- Worker Service → Checkpoint Control: reports acknowledgements
- Leader Election → Coordinator Store: coordinates leadership
- Leader Election → Kubernetes Leases: uses leases
- Binary Entrypoint → Metrics and Telemetry: initializes telemetry
- Worker Service → Metrics and Telemetry: records runtime metrics