How rishabhjain2003/distributedtaskqueuesystem works
1. What this repository is
- A small Python reference implementation of a distributed task queue built around Celery.
- It is split into a request/API layer and a background worker layer, with external infrastructure for message brokering, result storage, and monitoring.
2. Top-level service boundaries
- The repo contains two main Python packages:
- api/: the HTTP-facing service.
- dtq/: the Celery application and task runtime.
- Separate Dockerfiles exist for the API and worker images, which suggests they are deployed as distinct containers with different entrypoints.
3. API service responsibility
- api/main.py is the likely FastAPI application entrypoint.
- api/models.py likely holds request/response schemas.
- api/tasks.py, api/results.py, and api/utils.py suggest the API does three main things:
- accepts client requests,
- submits background jobs,
- retrieves job state/results.
4. Task submission flow
- The client talks only to the API over HTTP.
- The API does not perform the long-running computation itself; instead it publishes task messages to RabbitMQ.
- This is the main asynchronous boundary in the system: the HTTP request is decoupled from job execution.
5. Worker service responsibility
- dtq/tasks.py is the natural home for the actual Celery task functions executed by workers.
- Workers consume messages from RabbitMQ and run the long-running or resource-intensive computations.
- The README explicitly describes worker concurrency of 2 per worker instance, which means each worker container can execute multiple tasks in parallel.
6. Result propagation
- Task outcomes and task metadata are stored in Redis as the Celery result backend.
- The API can query Redis to return task status, success/failure, and final output to clients.
- This keeps the API stateless and avoids coupling request handling to task execution lifecycle.
7. Infrastructure services
- docker-compose.yml is the central orchestration file for the whole system.
- The architecture depends on:
- RabbitMQ on port 5672 for task delivery,
- Redis on port 6379 for result backend,
- Flower on port 5555 for task monitoring.
- The README’s diagram and setup instructions indicate the full stack is intended to run locally via Docker Compose.
8. Monitoring/operability
- Flower is an observability layer for Celery and RabbitMQ activity.
- It gives visibility into queued, active, succeeded, and failed tasks, which is important for a distributed system where workers may scale up/down or terminate.
9. Scalability and fault-tolerance model
- The design assumes workers are ephemeral and can terminate at any time, such as in autoscaled environments.
- Because execution state is externalized into RabbitMQ and Redis, the API and workers can scale independently.
- This is a classic “stateless control plane + scalable worker pool” architecture.
10. What an architecture diagram should emphasize
- Draw the API as the only client-facing component.
- Show a one-way enqueue path from API to RabbitMQ.
- Show multiple worker instances consuming from RabbitMQ in parallel.
- Show Redis as the shared result backend used by both workers and the API.
- Show Flower as a sidecar-like monitoring service connected to the broker/task system.
Main entry points: docker-compose.yml Compose · main.py API app · app.py Celery app · tasks.py Task funcs
Components 14
API service
HTTP-facing control plane that accepts requests, enqueues work, and exposes job state.
- API app · FastAPI-style entrypoint for the request layer.
api/main.py - Schemas · Request and response models used by the API.
api/models.py - Submit jobs · API-side code that turns HTTP requests into background tasks.
api/tasks.py - Job status · API-side code that reads task state and returns outcomes to clients.
api/results.py - Helpers · Shared API utilities used across request and result handling.
api/utils.py
Worker runtime
Celery execution layer that consumes queued jobs and performs the actual task work.
- Celery app · Worker-side Celery application configuration and wiring.
dtq/app.py - Task funcs · Executable Celery tasks run by worker processes.
dtq/tasks.py
Infrastructure
Shared broker, storage, monitoring, and local orchestration for the distributed system.
- RabbitMQ · Message broker used to deliver task messages to workers.
- Redis · Shared Celery result store for task metadata and outcomes.
- Flower · Celery monitoring UI for queue, worker, and task visibility.
- Compose · Docker Compose setup that assembles the API, workers, and infrastructure locally.
docker-compose.yml - API image · Docker build definition for the HTTP service.
Dockerfile.api - Worker image · Docker build definition for the Celery worker runtime.
Dockerfile.worker
Other components
- Client · External caller that submits jobs and checks status over HTTP.
Connections 22
- Client → API app: HTTP requests · Clients talk only to the API over HTTP.
- API app → Schemas: uses · API validates and serializes payloads with request/response schemas.
- API app → Submit jobs: submits · Request handling delegates job submission to the enqueue layer.
- API app → Job status: reads status · API resolves task state and results for clients.
- Submit jobs → RabbitMQ: enqueue · API publishes task messages to the broker.
- Job status → Redis: lookup · API reads task metadata and outcomes from the result backend.
- Submit jobs → Helpers: uses · Submission flow shares helper logic.
- Job status → Helpers: uses · Result retrieval shares helper logic.
- Celery app → RabbitMQ: consume · Celery connects to the broker for task delivery.
- Celery app → Redis: backend · Celery stores task results in Redis.
- Task funcs → Celery app: registered by · Task functions are bound into the Celery runtime.
- Task funcs → Redis: write results · Executed tasks persist success, failure, and output state.
- Task funcs → RabbitMQ: consume jobs · Workers pull queued work from RabbitMQ.
- Flower → RabbitMQ: observes · Flower monitors brokered task activity.
- Flower → Redis: inspects state · Flower reads task metadata to show execution status.
- Compose → API image: builds · Compose assembles the API container image.
- Compose → Worker image: builds · Compose assembles the worker container image.
- Compose → RabbitMQ: starts · Compose brings up the broker service.
- Compose → Redis: starts · Compose brings up the result backend.
- Compose → Flower: starts · Compose brings up monitoring.
- Compose → API app: runs · Compose launches the request service container.
- Compose → Celery app: runs · Compose launches worker containers from the Celery runtime.