# rishabhjain2003/distributedtaskqueuesystem architecture

> Architecture diagram of the GitHub repository rishabhjain2003/distributedtaskqueuesystem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishabhjain2003/distributedtaskqueuesystem
- Repository: https://github.com/rishabhjain2003/distributedtaskqueuesystem

## Overview

1. **What this repository is**
   - A small Python reference implementation of a **distributed task queue** built around **Celery**.
   - It is split into a **request/API layer** and a **background worker layer**, with external infrastructure for **message brokering**, **result storage**, and **monitoring**.

2. **Top-level service boundaries**
   - The repo contains two main Python packages:
     - `api/`: the HTTP-facing service.
     - `dtq/`: the Celery application and task runtime.
   - Separate Dockerfiles exist for the API and worker images, which suggests they are deployed as distinct containers with different entrypoints.

3. **API service responsibility**
   - `api/main.py` is the likely FastAPI application entrypoint.
   - `api/models.py` likely holds request/response schemas.
   - `api/tasks.py`, `api/results.py`, and `api/utils.py` suggest the API does three main things:
     - accepts client requests,
     - submits background jobs,
     - retrieves job state/results.

4. **Task submission flow**
   - The client talks only to the API over HTTP.
   - The API does not perform the long-running computation itself; instead it **publishes task messages to RabbitMQ**.
   - This is the main asynchronous boundary in the system: the HTTP request is decoupled from job execution.

5. **Worker service responsibility**
   - `dtq/tasks.py` is the natural home for the actual Celery task functions executed by workers.
   - Workers consume messages from RabbitMQ and run the long-running or resource-intensive computations.
   - The README explicitly describes worker concurrency of **2 per worker instance**, which means each worker container can execute multiple tasks in parallel.

6. **Result propagation**
   - Task outcomes and task metadata are stored in **Redis** as the Celery result backend.
   - The API can query Redis to return task status, success/failure, and final output to clients.
   - This keeps the API stateless and avoids coupling request handling to task execution lifecycle.

7. **Infrastructure services**
   - `docker-compose.yml` is the central orchestration file for the whole system.
   - The architecture depends on:
     - **RabbitMQ** on port `5672` for task delivery,
     - **Redis** on port `6379` for result backend,
     - **Flower** on port `5555` for task monitoring.
   - The README’s diagram and setup instructions indicate the full stack is intended to run locally via Docker Compose.

8. **Monitoring/operability**
   - Flower is an observability layer for Celery and RabbitMQ activity.
   - It gives visibility into queued, active, succeeded, and failed tasks, which is important for a distributed system where workers may scale up/down or terminate.

9. **Scalability and fault-tolerance model**
   - The design assumes workers are **ephemeral** and can terminate at any time, such as in autoscaled environments.
   - Because execution state is externalized into RabbitMQ and Redis, the API and workers can scale independently.
   - This is a classic “stateless control plane + scalable worker pool” architecture.

10. **What an architecture diagram should emphasize**
   - Draw the API as the only client-facing component.
   - Show a one-way enqueue path from API to RabbitMQ.
   - Show multiple worker instances consuming from RabbitMQ in parallel.
   - Show Redis as the shared result backend used by both workers and the API.
   - Show Flower as a sidecar-like monitoring service connected to the broker/task system.

Main entry points: [`docker-compose.yml`](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/docker-compose.yml) (Compose), [`api/main.py`](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/main.py) (API app), [`dtq/app.py`](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/app.py) (Celery app), [`dtq/tasks.py`](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/tasks.py) (Task funcs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api_service["API service"]
  node_api_main["API app<br/>http service<br/>[main.py]"]
  node_api_models["Schemas<br/>contracts<br/>[models.py]"]
  node_api_tasks["Submit jobs<br/>enqueue logic<br/>[tasks.py]"]
  node_api_results["Job status<br/>result lookup<br/>[results.py]"]
  node_api_utils["Helpers<br/>[utils.py]"]
end

subgraph group_worker_runtime["Worker runtime"]
  node_celery_app["Celery app<br/>task runtime<br/>[app.py]"]
  node_worker_tasks["Task funcs<br/>background tasks<br/>[tasks.py]"]
end

subgraph group_infra["Infrastructure"]
  node_rabbitmq["RabbitMQ<br/>broker"]
  node_redis[("Redis<br/>result backend")]
  node_flower(("Flower<br/>monitoring"))
  node_compose["Compose<br/>orchestration<br/>[docker-compose.yml]"]
  node_api_image["API image<br/>container build<br/>[Dockerfile.api]"]
  node_worker_image["Worker image<br/>container build<br/>[Dockerfile.worker]"]
end

node_client(("Client<br/>caller"))

node_client -->|"HTTP requests"| node_api_main
node_api_main -->|"uses"| node_api_models
node_api_main -->|"submits"| node_api_tasks
node_api_main -->|"reads status"| node_api_results
node_api_tasks -->|"enqueue"| node_rabbitmq
node_api_results -->|"lookup"| node_redis
node_api_tasks -->|"uses"| node_api_utils
node_api_results -->|"uses"| node_api_utils
node_celery_app -->|"consume"| node_rabbitmq
node_celery_app -->|"backend"| node_redis
node_worker_tasks -->|"registered by"| node_celery_app
node_worker_tasks -->|"write results"| node_redis
node_worker_tasks -->|"consume jobs"| node_rabbitmq
node_flower -->|"observes"| node_rabbitmq
node_flower -->|"inspects state"| node_redis
node_compose -->|"builds"| node_api_image
node_compose -->|"builds"| node_worker_image
node_compose -->|"starts"| node_rabbitmq
node_compose -->|"starts"| node_redis
node_compose -->|"starts"| node_flower
node_compose -->|"runs"| node_api_main
node_compose -->|"runs"| node_celery_app

click node_api_main "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/main.py"
click node_api_models "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/models.py"
click node_api_tasks "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/tasks.py"
click node_api_results "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/results.py"
click node_api_utils "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/utils.py"
click node_celery_app "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/app.py"
click node_worker_tasks "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/tasks.py"
click node_compose "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/docker-compose.yml"
click node_api_image "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/Dockerfile.api"
click node_worker_image "https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/Dockerfile.worker"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_main,node_api_models,node_api_tasks,node_api_results,node_api_utils toneBlue
class node_celery_app,node_worker_tasks toneAmber
class node_rabbitmq,node_redis,node_flower,node_compose,node_api_image,node_worker_image toneMint
class node_client toneNeutral
```

## Components

### API service

HTTP-facing control plane that accepts requests, enqueues work, and exposes job state.

- [API app](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/main.py): http service · FastAPI-style entrypoint for the request layer. · `api/main.py`
- [Schemas](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/models.py): contracts · Request and response models used by the API. · `api/models.py`
- [Submit jobs](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/tasks.py): enqueue logic · API-side code that turns HTTP requests into background tasks. · `api/tasks.py`
- [Job status](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/results.py): result lookup · API-side code that reads task state and returns outcomes to clients. · `api/results.py`
- [Helpers](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/api/utils.py): shared helpers · Shared API utilities used across request and result handling. · `api/utils.py`

### Worker runtime

Celery execution layer that consumes queued jobs and performs the actual task work.

- [Celery app](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/app.py): task runtime · Worker-side Celery application configuration and wiring. · `dtq/app.py`
- [Task funcs](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/dtq/tasks.py): background tasks · Executable Celery tasks run by worker processes. · `dtq/tasks.py`

### Infrastructure

Shared broker, storage, monitoring, and local orchestration for the distributed system.

- **RabbitMQ**: broker · Message broker used to deliver task messages to workers.
- **Redis**: result backend · Shared Celery result store for task metadata and outcomes.
- **Flower**: monitoring · Celery monitoring UI for queue, worker, and task visibility.
- [Compose](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/docker-compose.yml): orchestration · Docker Compose setup that assembles the API, workers, and infrastructure locally. · `docker-compose.yml`
- [API image](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/Dockerfile.api): container build · Docker build definition for the HTTP service. · `Dockerfile.api`
- [Worker image](https://github.com/rishabhjain2003/distributedtaskqueuesystem/blob/main/Dockerfile.worker): container build · Docker build definition for the Celery worker runtime. · `Dockerfile.worker`

### Other components

- **Client**: caller · External caller that submits jobs and checks status over HTTP.

## Connections

- Client → API app: HTTP requests · Clients talk only to the API over HTTP.
- API app → Schemas: uses · API validates and serializes payloads with request/response schemas.
- API app → Submit jobs: submits · Request handling delegates job submission to the enqueue layer.
- API app → Job status: reads status · API resolves task state and results for clients.
- Submit jobs → RabbitMQ: enqueue · API publishes task messages to the broker.
- Job status → Redis: lookup · API reads task metadata and outcomes from the result backend.
- Submit jobs → Helpers: uses · Submission flow shares helper logic.
- Job status → Helpers: uses · Result retrieval shares helper logic.
- Celery app → RabbitMQ: consume · Celery connects to the broker for task delivery.
- Celery app → Redis: backend · Celery stores task results in Redis.
- Task funcs → Celery app: registered by · Task functions are bound into the Celery runtime.
- Task funcs → Redis: write results · Executed tasks persist success, failure, and output state.
- Task funcs → RabbitMQ: consume jobs · Workers pull queued work from RabbitMQ.
- Flower → RabbitMQ: observes · Flower monitors brokered task activity.
- Flower → Redis: inspects state · Flower reads task metadata to show execution status.
- Compose → API image: builds · Compose assembles the API container image.
- Compose → Worker image: builds · Compose assembles the worker container image.
- Compose → RabbitMQ: starts · Compose brings up the broker service.
- Compose → Redis: starts · Compose brings up the result backend.
- Compose → Flower: starts · Compose brings up monitoring.
- Compose → API app: runs · Compose launches the request service container.
- Compose → Celery app: runs · Compose launches worker containers from the Celery runtime.
