# taskforcesh/bullmq architecture

> Architecture diagram of the GitHub repository taskforcesh/bullmq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/taskforcesh/bullmq
- Repository: https://github.com/taskforcesh/bullmq

## Overview

##### Main Components
- Core Library (BullMQ): This includes all the job and queue classes (found in src/classes and python/bullmq) that define the core functionality such as job scheduling, processing, error handling, and additional patterns (e.g., parent/child job dependencies, rate limiting).
- Redis Datastore: As the backbone of BullMQ, Redis is critical. Show it as a separate external component responsible for job storage, locks, events, and atomic operations.
- Worker and Processor Components: Illustrate separate workers (both Node.js and possibly Python workers) that pick jobs up from the queues. Emphasize the handling of job events, retries, and sandboxed process support.
- Queue Events & Telemetry: Identify modules for emitting and handling events, metrics, and telemetry (documented in the interfaces and telemetry areas). This is important for monitoring and debugging.
- Documentation, Configurations, and Testing: Although not runtime components, clearly note that there’s extensive documentation (docs folder), configuration (config folder, docker-compose.yml, etc.) and testing support (tests folder) that supports the development and deployment processes.

##### Relationships and Data Flow
- Producers (job creators) submit jobs to the Queue layer (via the Queue class).
- The Queue component interacts with Redis to enqueue job data using Lua scripts for atomic operations.
- Workers (via the Worker class) poll or subscribe for jobs, process them, and update their progress/status back in Redis.
- Queue Events are emitted during job completion or failure, and these events can be picked up by either a dedicated event handler (QueueEvents class) or external services for monitoring.
- Telemetry and metrics modules gather and send out performance data.
- If using BullMQ Pro features, add details for support like group rate limiting or batches.

##### Architectural Patterns and Design Principles
- Modular Design: The separation between the core job/queue functionality (in src/classes), the Lua commands (src/commands), and external Redis integration shows a clear separation of concerns.
- Extensibility: Include components showing how developers can extend or interact with the queue (e.g., through custom processors or error handling).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Producer Node
    Producer["Job Producer"]

    %% BullMQ Core Library - Node.js
    subgraph "BullMQ Core Library - Node.js"
        NodeJob["Job Class (src/classes/job.ts)"]
        NodeQueue["Queue Class (src/classes/queue.ts)"]
        NodeWorker["Worker Class (src/classes/worker.ts)"]
        ChildProcessor["Child Processor (src/classes/child-processor.ts)"]
        QueueEvents["Queue Events (src/classes/queue-events.ts)"]
        QueueEventsProd["Queue Events Producer (src/classes/queue-events-producer.ts)"]
    end

    %% BullMQ Core Library - Python
    subgraph "BullMQ Core Library - Python"
        PyJob["Job (python/bullmq/job.py)"]
        PyQueue["Queue (python/bullmq/queue.py)"]
        PyWorker["Worker (python/bullmq/worker.py)"]
    end

    %% Lua Scripting Layer
    subgraph "Lua Scripting Layer"
        LuaScript["Lua Script: addDelayedJob-6.lua"]
        LuaIncludes["Lua Includes (src/commands/includes)"]
    end

    %% External Redis Datastore
    Redis["Redis – Data Store"]

    %% Telemetry & Metrics
    subgraph "Telemetry & Metrics"
        TelemetryDocs["Telemetry Docs (docs/gitbook/guide/telemetry/README.md)"]
        TelemetryEnum["Telemetry Enum (src/enums/telemetry-attributes.ts)"]
        TelemetryIntf["Telemetry Interface (src/interfaces/telemetry.ts)"]
    end

    %% Support Components
    subgraph "Support"
        Docs["Documentation (docs)"]
        Config["Configurations (config & docker-compose.yml)"]
        Tests["Testing (tests)"]
        PyTests["Python Tests (python/tests)"]
    end

    %% Connections

    %% Producers submit jobs to the Queue classes
    Producer -->|"submits_job"| NodeQueue
    Producer -->|"submits_job"| PyQueue

    %% Queue classes use Lua scripts for enqueuing jobs
    NodeQueue -->|"enqueue_job"| LuaScript
    PyQueue -->|"enqueue_job"| LuaScript

    %% Lua scripts interact with Redis
    LuaScript -->|"atomic_exec"| Redis
    LuaIncludes -->|"supports"| LuaScript

    %% Redis dispatches jobs to Workers
    Redis -->|"job_dispatch"| NodeWorker
    Redis -->|"job_dispatch"| PyWorker

    %% Workers update status and emit events
    NodeWorker -->|"job_complete"| QueueEvents
    PyWorker -->|"job_complete"| QueueEvents

    %% Queue Events and Producer interaction
    QueueEventsProd -->|"produce_event"| QueueEvents

    %% Workers may delegate child tasks
    NodeWorker -->|"handles_child"| ChildProcessor

    %% Queue Events notify Telemetry components
    QueueEvents -->|"notify"| TelemetryDocs
    QueueEvents -->|"notify"| TelemetryEnum
    QueueEvents -->|"notify"| TelemetryIntf

    %% Styling
    classDef core fill:#00FFFF,stroke:#000,stroke-width:2px;
    classDef redis fill:#FF4500,stroke:#000,stroke-width:2px;
    classDef lua fill:#9400D3,stroke:#000,stroke-width:2px;
    classDef worker fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef events fill:#FF00FF,stroke:#000,stroke-width:2px;
    classDef support fill:#ADD8E6,stroke:#000,stroke-width:2px;

    class NodeJob,NodeQueue,ChildProcessor,QueueEvents,QueueEventsProd,PyJob,PyQueue core;
    class LuaScript,LuaIncludes lua;
    class NodeWorker,PyWorker worker;
    class QueueEvents events;
    class Redis redis;
    class TelemetryDocs,TelemetryEnum,TelemetryIntf support;
    class Docs,Config,Tests,PyTests support;

    %% Click Events
    click NodeJob "https://github.com/taskforcesh/bullmq/blob/master/src/classes/job.ts"
    click NodeQueue "https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue.ts"
    click NodeWorker "https://github.com/taskforcesh/bullmq/blob/master/src/classes/worker.ts"
    click ChildProcessor "https://github.com/taskforcesh/bullmq/blob/master/src/classes/child-processor.ts"
    click QueueEvents "https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue-events.ts"
    click QueueEventsProd "https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue-events-producer.ts"
    click PyJob "https://github.com/taskforcesh/bullmq/blob/master/python/bullmq/job.py"
    click PyQueue "https://github.com/taskforcesh/bullmq/blob/master/python/bullmq/queue.py"
    click PyWorker "https://github.com/taskforcesh/bullmq/blob/master/python/bullmq/worker.py"
    click LuaScript "https://github.com/taskforcesh/bullmq/blob/master/src/commands/addDelayedJob-6.lua"
    click LuaIncludes "https://github.com/taskforcesh/bullmq/tree/master/src/commands/includes"
    click TelemetryDocs "https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/guide/telemetry/README.md"
    click TelemetryEnum "https://github.com/taskforcesh/bullmq/blob/master/src/enums/telemetry-attributes.ts"
    click TelemetryIntf "https://github.com/taskforcesh/bullmq/blob/master/src/interfaces/telemetry.ts"
    click Docs "https://github.com/taskforcesh/bullmq/tree/master/docs"
    click Config "https://github.com/taskforcesh/bullmq/tree/master/config"
    click Tests "https://github.com/taskforcesh/bullmq/tree/master/tests"
    click PyTests "https://github.com/taskforcesh/bullmq/tree/master/python/tests"
```
