# pulkit-999/task-processing-platform architecture

> Architecture diagram of the GitHub repository pulkit-999/task-processing-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pulkit-999/task-processing-platform
- Repository: https://github.com/pulkit-999/task-processing-platform

## Overview

1. **Top-level shape:** This repo is split into two runnable services plus shared internal packages: `cmd/api` is the producer-facing HTTP API, and `cmd/worker` is the background consumer. The shared logic lives under `internal/`.

2. **Request ingress:** The API service is the only synchronous entry point. The README indicates it uses **Gin** and exposes at least `POST /job` to accept work and `GET /metrics` to report system state.

3. **Job contract:** `internal/models/job.go` defines the job payload that crosses service boundaries. From the README, each job carries a `type`, `payload`, and `priority`, so this model is the canonical schema shared by API and worker.

4. **Queue boundary:** `internal/queue/redis_queue.go` is the infrastructure adapter around **Redis**. This is the main persistence/transport boundary in the system: API enqueues jobs, workers dequeue them, and Redis holds pending work between producers and consumers.

5. **Prioritization:** Priority scheduling is part of the queueing layer rather than the HTTP layer. The architecture implies separate queueing behavior for high/medium/low priority jobs, with Redis acting as the backing store for that ordering.

6. **Background execution:** `internal/worker/worker..go` (likely the worker pool implementation) is the processing engine. It runs concurrent goroutines to pull jobs from Redis and execute them independently of the API request path.

7. **Reliability path:** Failed jobs are not dropped immediately. The worker layer handles retries with exponential backoff, and after the retry limit is reached, jobs are moved to a **dead-letter queue (DLQ)**, which is part of the failure-handling boundary.

8. **Metrics subsystem:** `internal/metrics/metrics.go` tracks operational counts such as processed, failed, and queued jobs. The API exposes those counters through `GET /metrics`, so observability is built into the service layer rather than delegated to an external monitoring stack.

9. **Runtime/deployment model:** `Dockerfile` and `docker-compose.yml` show that the system is intended to run as a containerized multi-service deployment. The important runtime dependency is **Redis**, which must be available to both the API and worker containers.

10. **End-to-end data flow:** Client submits a job to the API → API validates/maps it into the job model → job is pushed into Redis → worker pool consumes and processes it concurrently → failures are retried or sent to the DLQ → metrics are updated and exposed through the API.

11. **Key architectural boundaries:** The system cleanly separates synchronous API concerns from asynchronous execution concerns, and isolates all queue semantics behind the Redis queue adapter. That makes the main diagram blocks: client, API service, Redis queue/DLQ, worker pool, and metrics exposure.

Main entry points: [`internal/worker/worker..go`](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/worker/worker..go) (Worker pool), [`cmd/api/main.go`](https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/api/main.go) (API main), [`internal/queue/redis_queue.go`](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/queue/redis_queue.go) (Redis queue), [`internal/metrics/metrics.go`](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/metrics/metrics.go) (Metrics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_services["Services"]
  node_node_api_main["API main<br/>entrypoint<br/>[main.go]"]
  node_node_api_routes["Job API<br/>http api<br/>[main.go]"]
  node_node_worker_main["Worker main<br/>entrypoint<br/>[main.go]"]
end

subgraph group_group_shared["Shared internals"]
  node_node_worker_pool["Worker pool<br/>processor<br/>[worker..go]"]
  node_node_job_model["Job model<br/>schema<br/>[job.go]"]
  node_node_queue_adapter[("Redis queue<br/>queue adapter<br/>[redis_queue.go]")]
  node_node_metrics(("Metrics<br/>telemetry<br/>[metrics.go]"))
end

subgraph group_group_runtime["Runtime"]
  node_node_redis[("Redis<br/>external store")]
  node_node_dockerfile["Dockerfile<br/>container build"]
  node_node_compose["Compose<br/>deployment<br/>[docker-compose.yml]"]
end

node_node_client(("Client<br/>producer"))

node_node_client -->|"POST /job"| node_node_api_routes
node_node_api_main -->|"hosts"| node_node_api_routes
node_node_api_main -->|"enqueue"| node_node_queue_adapter
node_node_api_main -->|"reads"| node_node_metrics
node_node_api_routes -->|"maps to"| node_node_job_model
node_node_worker_main -->|"starts"| node_node_worker_pool
node_node_worker_pool -->|"dequeue"| node_node_queue_adapter
node_node_worker_pool -->|"consumes"| node_node_job_model
node_node_worker_pool -->|"updates"| node_node_metrics
node_node_queue_adapter -->|"backed by"| node_node_redis
node_node_queue_adapter -->|"reports"| node_node_metrics
node_node_worker_pool -->|"dlq/retry"| node_node_redis
node_node_compose -->|"runs"| node_node_api_main
node_node_compose -->|"runs"| node_node_worker_main
node_node_compose -->|"includes"| node_node_redis
node_node_dockerfile -->|"builds"| node_node_api_main
node_node_dockerfile -->|"builds"| node_node_worker_main
node_node_client -.->|"observes"| node_node_metrics

click node_node_api_main "https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/api/main.go"
click node_node_api_routes "https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/api/main.go"
click node_node_worker_main "https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/worker/main.go"
click node_node_worker_pool "https://github.com/pulkit-999/task-processing-platform/blob/main/internal/worker/worker..go"
click node_node_job_model "https://github.com/pulkit-999/task-processing-platform/blob/main/internal/models/job.go"
click node_node_queue_adapter "https://github.com/pulkit-999/task-processing-platform/blob/main/internal/queue/redis_queue.go"
click node_node_metrics "https://github.com/pulkit-999/task-processing-platform/blob/main/internal/metrics/metrics.go"
click node_node_dockerfile "https://github.com/pulkit-999/task-processing-platform/tree/main/Dockerfile"
click node_node_compose "https://github.com/pulkit-999/task-processing-platform/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_api_main,node_node_api_routes,node_node_worker_main toneBlue
class node_node_worker_pool,node_node_job_model,node_node_queue_adapter,node_node_metrics toneAmber
class node_node_redis,node_node_dockerfile,node_node_compose toneMint
class node_node_client toneNeutral
```

## Components

### Services

Runnable entrypoints for API and background work

- [API main](https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/api/main.go): entrypoint · Starts the HTTP server and wires the API service · `cmd/api/main.go`
- [Job API](https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/api/main.go): http api · Accepts jobs and exposes metrics endpoints · `cmd/api/main.go`
- [Worker main](https://github.com/pulkit-999/task-processing-platform/blob/main/cmd/worker/main.go): entrypoint · Starts the background consumer process · `cmd/worker/main.go`

### Shared internals

Common domain, queue, metrics, and worker logic

- [Worker pool](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/worker/worker..go): processor · Concurrent job execution and retry handling · `internal/worker/worker..go`
- [Job model](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/models/job.go): schema · Canonical job contract shared across services · `internal/models/job.go`
- [Redis queue](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/queue/redis_queue.go): queue adapter · Enqueues, dequeues, prioritizes, and DLQ-handles jobs · `internal/queue/redis_queue.go`
- [Metrics](https://github.com/pulkit-999/task-processing-platform/blob/main/internal/metrics/metrics.go): telemetry · Tracks queued, processed, and failed job counts · `internal/metrics/metrics.go`

### Runtime

Containerized deployment and external dependency

- **Redis**: external store · Backs the queue, priority ordering, and dead-letter storage
- [Dockerfile](https://github.com/pulkit-999/task-processing-platform/tree/main/Dockerfile): container build · Builds the application image · `Dockerfile`
- [Compose](https://github.com/pulkit-999/task-processing-platform/blob/main/docker-compose.yml): deployment · Runs API, worker, and Redis together · `docker-compose.yml`

### Other components

- **Client**: producer · Submits jobs to the HTTP API

## Connections

- Client → Job API: POST /job · Submits work requests
- API main → Job API: hosts · Bootstraps the HTTP service
- API main → Redis queue: enqueue · Sends accepted jobs to the queue
- API main → Metrics: reads · Serves current counters via metrics endpoint
- Job API → Job model: maps to · Validates and converts request payloads into the shared job schema
- Worker main → Worker pool: starts · Launches the background processing engine
- Worker pool → Redis queue: dequeue · Pulls jobs from the queue for processing
- Worker pool → Job model: consumes · Processes the shared job contract
- Worker pool → Metrics: updates · Records processed and failed job counts
- Redis queue → Redis: backed by · Uses Redis for queue persistence and ordering
- Redis queue → Metrics: reports · Updates queued job counts and queue state
- Worker pool → Redis: dlq/retry · Uses Redis-backed retry and dead-letter paths
- Compose → API main: runs · Defines the API container
- Compose → Worker main: runs · Defines the worker container
- Compose → Redis: includes · Provides the Redis service
- Dockerfile → API main: builds · Builds the application runtime image
- Dockerfile → Worker main: builds · Builds the worker runtime image
- Client → Metrics: observes · Can inspect system state through metrics exposure
