How pulkit-999/task-processing-platform works
1. Top-level shape: This repo is split into two runnable services plus shared internal packages: cmd/api is the producer-facing HTTP API, and cmd/worker is the background consumer. The shared logic lives under internal/.
2. Request ingress: The API service is the only synchronous entry point. The README indicates it uses Gin and exposes at least POST /job to accept work and GET /metrics to report system state.
3. Job contract: internal/models/job.go defines the job payload that crosses service boundaries. From the README, each job carries a type, payload, and priority, so this model is the canonical schema shared by API and worker.
4. Queue boundary: internal/queue/redis_queue.go is the infrastructure adapter around Redis. This is the main persistence/transport boundary in the system: API enqueues jobs, workers dequeue them, and Redis holds pending work between producers and consumers.
5. Prioritization: Priority scheduling is part of the queueing layer rather than the HTTP layer. The architecture implies separate queueing behavior for high/medium/low priority jobs, with Redis acting as the backing store for that ordering.
6. Background execution: internal/worker/worker..go (likely the worker pool implementation) is the processing engine. It runs concurrent goroutines to pull jobs from Redis and execute them independently of the API request path.
7. Reliability path: Failed jobs are not dropped immediately. The worker layer handles retries with exponential backoff, and after the retry limit is reached, jobs are moved to a dead-letter queue (DLQ), which is part of the failure-handling boundary.
8. Metrics subsystem: internal/metrics/metrics.go tracks operational counts such as processed, failed, and queued jobs. The API exposes those counters through GET /metrics, so observability is built into the service layer rather than delegated to an external monitoring stack.
9. Runtime/deployment model: Dockerfile and docker-compose.yml show that the system is intended to run as a containerized multi-service deployment. The important runtime dependency is Redis, which must be available to both the API and worker containers.
10. End-to-end data flow: Client submits a job to the API → API validates/maps it into the job model → job is pushed into Redis → worker pool consumes and processes it concurrently → failures are retried or sent to the DLQ → metrics are updated and exposed through the API.
11. Key architectural boundaries: The system cleanly separates synchronous API concerns from asynchronous execution concerns, and isolates all queue semantics behind the Redis queue adapter. That makes the main diagram blocks: client, API service, Redis queue/DLQ, worker pool, and metrics exposure.
Main entry points: worker..go Worker pool · main.go API main · redis_queue.go Redis queue · metrics.go Metrics
Components 11
Services
Runnable entrypoints for API and background work
- API main · Starts the HTTP server and wires the API service
cmd/api/main.go - Job API · Accepts jobs and exposes metrics endpoints
cmd/api/main.go - Worker main · Starts the background consumer process
cmd/worker/main.go
Shared internals
Common domain, queue, metrics, and worker logic
- Worker pool · Concurrent job execution and retry handling
internal/worker/worker..go - Job model · Canonical job contract shared across services
internal/models/job.go - Redis queue · Enqueues, dequeues, prioritizes, and DLQ-handles jobs
internal/queue/redis_queue.go - Metrics · Tracks queued, processed, and failed job counts
internal/metrics/metrics.go
Runtime
Containerized deployment and external dependency
- Redis · Backs the queue, priority ordering, and dead-letter storage
- Dockerfile · Builds the application image
Dockerfile - Compose · Runs API, worker, and Redis together
docker-compose.yml
Other components
- Client · Submits jobs to the HTTP API
Connections 18
- Client → Job API: POST /job · Submits work requests
- API main → Job API: hosts · Bootstraps the HTTP service
- API main → Redis queue: enqueue · Sends accepted jobs to the queue
- API main → Metrics: reads · Serves current counters via metrics endpoint
- Job API → Job model: maps to · Validates and converts request payloads into the shared job schema
- Worker main → Worker pool: starts · Launches the background processing engine
- Worker pool → Redis queue: dequeue · Pulls jobs from the queue for processing
- Worker pool → Job model: consumes · Processes the shared job contract
- Worker pool → Metrics: updates · Records processed and failed job counts
- Redis queue → Redis: backed by · Uses Redis for queue persistence and ordering
- Redis queue → Metrics: reports · Updates queued job counts and queue state
- Worker pool → Redis: dlq/retry · Uses Redis-backed retry and dead-letter paths
- Compose → API main: runs · Defines the API container
- Compose → Worker main: runs · Defines the worker container
- Compose → Redis: includes · Provides the Redis service
- Dockerfile → API main: builds · Builds the application runtime image
- Dockerfile → Worker main: builds · Builds the worker runtime image
- Client → Metrics: observes · Can inspect system state through metrics exposure