# heydeepakch/distributed-task-scheduler architecture

> Architecture diagram of the GitHub repository heydeepakch/distributed-task-scheduler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/heydeepakch/distributed-task-scheduler
- Repository: https://github.com/heydeepakch/distributed-task-scheduler

## Overview

#### Purpose and Entry Points
This Go repository is a distributed asynchronous task scheduler. Producers serialize typed jobs and enqueue them in Redis; consumer nodes atomically claim, rate-limit, execute, acknowledge, and recover jobs. Primary executables are `producer/main.go` (task submission) and `consumer/main.go` (worker node). The task contract is `proto/task.proto`, with generated bindings in `proto/task.pb.go`.

#### Task Ingestion and Contract
The producer constructs `proto.Task` records, protobuf-serializes them, and pushes bytes to the pending Redis list. The protobuf schema is the compatibility boundary between independently deployed producer and consumer processes.
- `producer/main.go`
- `proto/task.proto`
- `proto/task.pb.go`

#### Redis Queue State Machine
Redis is the durable broker and source of distributed coordination. Jobs transition from a pending queue to a processing queue through blocking atomic list movement (`BRPopLPush`/reliable-queue pattern). This prevents loss between dequeue and worker handoff, establishing at-least-once delivery; downstream effects must therefore be idempotent.
- `consumer/main.go`
- `producer/main.go`
- `config/config.go`

#### Consumer Dispatch and Bounded Execution
The consumer dispatch loop deserializes claimed tasks and sends them to a bounded Go channel. A fixed worker pool consumes this channel, making channel blocking the backpressure boundary: Redis claiming slows when execution capacity is exhausted. Execution-specific business behavior is isolated from queue orchestration.
- `consumer/main.go`
- `worker/executor.go`
- `config/config.go`

#### Completion, Failure, and Recovery
Workers execute tasks and remove successfully completed serialized payloads from the processing list (`LREM`). Failed, rate-limited, or crash-interrupted jobs remain visible in processing. A periodic reaper moves processing items back to pending, supplying retries and recovery but also allowing duplicate execution.
- `consumer/main.go`
- `worker/executor.go`
- `config/config.go`

#### Distributed Rate-Limit Gate
Before selected task types execute, workers use a Redis-side Lua fixed-window counter. The Go limiter invokes the script; Redis executes check/increment/TTL operations atomically across all consumer replicas. A denied task is intentionally left for reaper-driven retry rather than acknowledged.
- `pkg/ratelimiter/ratelimiter.go`
- `lua/rate_limit.lua`
- `consumer/main.go`

#### Observability and Deployment Boundaries
Consumer nodes expose Prometheus metrics for backlog, execution duration, and errors. Prometheus scrapes the metrics endpoint using repository configuration. Runtime infrastructure is Redis 7+ for queues and rate-limit state, Prometheus for monitoring, and Docker for a compact statically built consumer image.
- `metrics/metrics.go`
- `prometheus.yml`
- `Dockerfile`

Main entry points: [`consumer/main.go`](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go) (Consumer dispatcher), [`worker/executor.go`](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/worker/executor.go) (Worker pool and executor), [`proto/task.pb.go`](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/proto/task.pb.go) (Generated task bindings), [`producer/main.go`](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/producer/main.go) (Producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_producer["Task submission"]
  node_producer_main["Producer<br/>Go executable<br/>[main.go]"]
  node_task_schema["Task contract<br/>Protobuf schema<br/>[task.proto]"]
  node_task_bindings["Generated task bindings<br/>Go protobuf<br/>[task.pb.go]"]
end

subgraph group_consumer["Consumer node"]
  node_consumer_main["Consumer dispatcher<br/>Go executable<br/>[main.go]"]
  node_work_channel["Bounded work channel<br/>Backpressure boundary<br/>[main.go]"]
  node_worker_executor["Worker pool and executor<br/>Go workers<br/>[executor.go]"]
  node_rate_limiter["Distributed rate limiter<br/>Go limiter<br/>[ratelimiter.go]"]
  node_reaper["Recovery reaper<br/>Consumer loop<br/>[main.go]"]
  node_metrics["Consumer metrics<br/>Prometheus instrumentation<br/>[metrics.go]"]
end

subgraph group_infra["Runtime infrastructure"]
  node_pending_queue["Pending queue<br/>Redis list"]
  node_processing_queue["Processing queue<br/>Redis list"]
  node_redis[("Redis 7+<br/>Broker and coordination")]
  node_rate_limit_lua["Atomic rate-limit script<br/>Redis Lua<br/>[rate_limit.lua]"]
  node_prometheus["Prometheus<br/>Monitoring server<br/>[prometheus.yml]"]
  node_consumer_image["Consumer image<br/>Docker build"]
end

node_task_schema -->|"generates"| node_task_bindings
node_producer_main -->|"constructs and serializes"| node_task_bindings
node_producer_main -->|"enqueues payloads"| node_pending_queue
node_pending_queue -->|"stored in"| node_redis
node_processing_queue -->|"stored in"| node_redis
node_consumer_main -->|"atomically claims from"| node_pending_queue
node_consumer_main -->|"moves claimed payload to"| node_processing_queue
node_consumer_main -->|"deserializes"| node_task_bindings
node_consumer_main -->|"dispatches to"| node_work_channel
node_work_channel -->|"consumed by"| node_worker_executor
node_worker_executor -->|"checks selected tasks"| node_rate_limiter
node_rate_limiter -->|"invokes"| node_rate_limit_lua
node_rate_limit_lua -->|"atomically updates counters"| node_redis
node_worker_executor -->|"LREM on success"| node_processing_queue
node_reaper -->|"scans retained jobs"| node_processing_queue
node_reaper -->|"requeues for retry"| node_pending_queue
node_consumer_main -->|"records queue and execution signals"| node_metrics
node_prometheus -->|"scrapes"| node_metrics
node_consumer_image -->|"packages"| node_consumer_main

click node_producer_main "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/producer/main.go"
click node_task_schema "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/proto/task.proto"
click node_task_bindings "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/proto/task.pb.go"
click node_consumer_main "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go"
click node_work_channel "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go"
click node_worker_executor "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/worker/executor.go"
click node_rate_limiter "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/pkg/ratelimiter/ratelimiter.go"
click node_rate_limit_lua "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/lua/rate_limit.lua"
click node_reaper "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go"
click node_metrics "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/metrics/metrics.go"
click node_prometheus "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/prometheus.yml"
click node_consumer_image "https://github.com/heydeepakch/distributed-task-scheduler/blob/main/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer_main,node_task_schema,node_task_bindings toneBlue
class node_consumer_main,node_work_channel,node_worker_executor,node_rate_limiter,node_reaper,node_metrics toneAmber
class node_pending_queue,node_processing_queue,node_redis,node_rate_limit_lua,node_prometheus,node_consumer_image toneMint
```

## Components

### Task submission

- [Producer](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/producer/main.go): Go executable · Serializes and submits typed tasks. · `producer/main.go`
- [Task contract](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/proto/task.proto): Protobuf schema · Compatibility boundary between producer and consumer deployments. · `proto/task.proto`
- [Generated task bindings](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/proto/task.pb.go): Go protobuf · `proto/task.pb.go`

### Consumer node

- [Consumer dispatcher](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go): Go executable · Atomically claims tasks and feeds bounded execution. · `consumer/main.go`
- [Bounded work channel](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go): Backpressure boundary · Blocks further Redis claiming when workers are saturated. · `consumer/main.go`
- [Worker pool and executor](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/worker/executor.go): Go workers · Executes task-specific behavior and acknowledges successes. · `worker/executor.go`
- [Distributed rate limiter](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/pkg/ratelimiter/ratelimiter.go): Go limiter · Gates selected task types before execution. · `pkg/ratelimiter/ratelimiter.go`
- [Recovery reaper](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/consumer/main.go): Consumer loop · Returns retained processing items to pending for retry and crash recovery. · `consumer/main.go`
- [Consumer metrics](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/metrics/metrics.go): Prometheus instrumentation · Publishes backlog, execution duration, and error metrics. · `metrics/metrics.go`

### Runtime infrastructure

- **Pending queue**: Redis list · Durable queue of serialized tasks awaiting claim.
- **Processing queue**: Redis list · Visible in-flight payloads retained until acknowledgement.
- **Redis 7+**: Broker and coordination · Hosts queue state and distributed rate-limit counters.
- [Atomic rate-limit script](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/lua/rate_limit.lua): Redis Lua · Implements fixed-window check, increment, and TTL atomically. · `lua/rate_limit.lua`
- [Prometheus](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/prometheus.yml): Monitoring server · Scrapes the consumer metrics endpoint. · `prometheus.yml`
- [Consumer image](https://github.com/heydeepakch/distributed-task-scheduler/blob/main/Dockerfile): Docker build · Compact statically built consumer runtime image. · `Dockerfile`

## Connections

- Task contract → Generated task bindings: generates
- Producer → Generated task bindings: constructs and serializes
- Producer → Pending queue: enqueues payloads
- Pending queue → Redis 7+: stored in
- Processing queue → Redis 7+: stored in
- Consumer dispatcher → Pending queue: atomically claims from · Uses blocking reliable queue movement.
- Consumer dispatcher → Processing queue: moves claimed payload to
- Consumer dispatcher → Generated task bindings: deserializes
- Consumer dispatcher → Bounded work channel: dispatches to
- Bounded work channel → Worker pool and executor: consumed by
- Worker pool and executor → Distributed rate limiter: checks selected tasks
- Distributed rate limiter → Atomic rate-limit script: invokes
- Atomic rate-limit script → Redis 7+: atomically updates counters
- Worker pool and executor → Processing queue: LREM on success · Failures and denied jobs remain for retry.
- Recovery reaper → Processing queue: scans retained jobs
- Recovery reaper → Pending queue: requeues for retry
- Consumer dispatcher → Consumer metrics: records queue and execution signals
- Prometheus → Consumer metrics: scrapes
- Consumer image → Consumer dispatcher: packages