How heydeepakch/distributed-task-scheduler works
Purpose and Entry Points
This Go repository is a distributed asynchronous task scheduler. Producers serialize typed jobs and enqueue them in Redis; consumer nodes atomically claim, rate-limit, execute, acknowledge, and recover jobs. Primary executables are producer/main.go (task submission) and consumer/main.go (worker node). The task contract is proto/task.proto, with generated bindings in proto/task.pb.go.
Task Ingestion and Contract
The producer constructs proto.Task records, protobuf-serializes them, and pushes bytes to the pending Redis list. The protobuf schema is the compatibility boundary between independently deployed producer and consumer processes.
• producer/main.go
• proto/task.proto
• proto/task.pb.go
Redis Queue State Machine
Redis is the durable broker and source of distributed coordination. Jobs transition from a pending queue to a processing queue through blocking atomic list movement (BRPopLPush/reliable-queue pattern). This prevents loss between dequeue and worker handoff, establishing at-least-once delivery; downstream effects must therefore be idempotent.
• consumer/main.go
• producer/main.go
• config/config.go
Consumer Dispatch and Bounded Execution
The consumer dispatch loop deserializes claimed tasks and sends them to a bounded Go channel. A fixed worker pool consumes this channel, making channel blocking the backpressure boundary: Redis claiming slows when execution capacity is exhausted. Execution-specific business behavior is isolated from queue orchestration.
• consumer/main.go
• worker/executor.go
• config/config.go
Completion, Failure, and Recovery
Workers execute tasks and remove successfully completed serialized payloads from the processing list (LREM). Failed, rate-limited, or crash-interrupted jobs remain visible in processing. A periodic reaper moves processing items back to pending, supplying retries and recovery but also allowing duplicate execution.
• consumer/main.go
• worker/executor.go
• config/config.go
Distributed Rate-Limit Gate
Before selected task types execute, workers use a Redis-side Lua fixed-window counter. The Go limiter invokes the script; Redis executes check/increment/TTL operations atomically across all consumer replicas. A denied task is intentionally left for reaper-driven retry rather than acknowledged.
• pkg/ratelimiter/ratelimiter.go
• lua/rate_limit.lua
• consumer/main.go
Observability and Deployment Boundaries
Consumer nodes expose Prometheus metrics for backlog, execution duration, and errors. Prometheus scrapes the metrics endpoint using repository configuration. Runtime infrastructure is Redis 7+ for queues and rate-limit state, Prometheus for monitoring, and Docker for a compact statically built consumer image.
• metrics/metrics.go
• prometheus.yml
• Dockerfile
Main entry points: consumer/main.go Consumer dispatcher · executor.go Worker pool and executor · task.pb.go Generated task bindings · producer/main.go Producer
Components 15
Task submission
- Producer · Serializes and submits typed tasks.
producer/main.go - Task contract · Compatibility boundary between producer and consumer deployments.
proto/task.proto - Generated task bindings
proto/task.pb.go
Consumer node
- Consumer dispatcher · Atomically claims tasks and feeds bounded execution.
consumer/main.go - Bounded work channel · Blocks further Redis claiming when workers are saturated.
consumer/main.go - Worker pool and executor · Executes task-specific behavior and acknowledges successes.
worker/executor.go - Distributed rate limiter · Gates selected task types before execution.
pkg/ratelimiter/ratelimiter.go - Recovery reaper · Returns retained processing items to pending for retry and crash recovery.
consumer/main.go - Consumer metrics · Publishes backlog, execution duration, and error metrics.
metrics/metrics.go
Runtime infrastructure
- Pending queue · Durable queue of serialized tasks awaiting claim.
- Processing queue · Visible in-flight payloads retained until acknowledgement.
- Redis 7+ · Hosts queue state and distributed rate-limit counters.
- Atomic rate-limit script · Implements fixed-window check, increment, and TTL atomically.
lua/rate_limit.lua - Prometheus · Scrapes the consumer metrics endpoint.
prometheus.yml - Consumer image · Compact statically built consumer runtime image.
Dockerfile
Connections 19
- Task contract → Generated task bindings: generates
- Producer → Generated task bindings: constructs and serializes
- Producer → Pending queue: enqueues payloads
- Pending queue → Redis 7+: stored in
- Processing queue → Redis 7+: stored in
- Consumer dispatcher → Pending queue: atomically claims from · Uses blocking reliable queue movement.
- Consumer dispatcher → Processing queue: moves claimed payload to
- Consumer dispatcher → Generated task bindings: deserializes
- Consumer dispatcher → Bounded work channel: dispatches to
- Bounded work channel → Worker pool and executor: consumed by
- Worker pool and executor → Distributed rate limiter: checks selected tasks
- Distributed rate limiter → Atomic rate-limit script: invokes
- Atomic rate-limit script → Redis 7+: atomically updates counters
- Worker pool and executor → Processing queue: LREM on success · Failures and denied jobs remain for retry.
- Recovery reaper → Processing queue: scans retained jobs
- Recovery reaper → Pending queue: requeues for retry
- Consumer dispatcher → Consumer metrics: records queue and execution signals
- Prometheus → Consumer metrics: scrapes
- Consumer image → Consumer dispatcher: packages