# bogdanp/dramatiq architecture

> Architecture diagram of the GitHub repository bogdanp/dramatiq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bogdanp/dramatiq
- Repository: https://github.com/bogdanp/dramatiq

## Overview

##### Project type and purpose
- Dramatiq is an open-source Python library for distributed task processing (a lightweight Celery alternative).
- Its core purpose is to let application code define “actors” (tasks), send messages into a broker (RabbitMQ, Redis, or stub), and have worker processes consume and execute those tasks reliably, with support for middleware, results backends, and rate limiting.

##### File structure
- Top-level directories
– dramatiq/: the core library code, broken into submodules (actor, broker, worker, middleware, rate_limits, results, etc.)
– examples/: sample apps demonstrating usage patterns
– benchmarks/: performance tests
– tests/: unit and integration tests
- Key files
– dramatiq/cli.py and __main__.py: entry points for the dramatiq command-line tool
– dramatiq/brokers/: implementations for RabbitMQ, Redis, stub, callable
– dramatiq/worker.py & watcher.py: worker loop and file-watcher for auto-reload
– dramatiq/middleware/: pluggable behavior chain (retries, timeouts, metrics, etc.)
– dramatiq/results/ and dramatiq/rate_limits/: optional backends and middleware
- Configuration & build
– setup.py, setup.cfg, MANIFEST.in: packaging

##### README for additional insights
- Quickstart shows how a user:
– Defines an @dramatiq.actor in application code
– Invokes actor.send(...) which publishes a message to the configured broker
– Launches dramatq workers via CLI to consume and run tasks
- README highlights external dependencies: RabbitMQ or Redis for brokers, optional results/watch support

a. Main components
– Application / Producer: Python code defining actors and enqueuing messages
– CLI / Launcher: ‘dramatiq’ command that starts Worker(s) or Watcher
– Broker: external queue service (RabbitMQ, Redis) or in-process stub
– Worker Process: the dramatiq.worker that:
- Polls the broker for messages
- Passes each message through the middleware pipeline
- Invokes the target actor function
- (Optionally) writes results to a results backend
– Middleware Pipeline: ordered modules (retries, time_limit, metrics, rate_limits, etc.)
– Results Backend: Redis, Memcached, or stub for storing/reading task results
– Rate Limiters: concurrency & throughput controls implemented as middleware
– Canteen (composition): support for grouping and orchestrating actors
– Watcher: optional file-change monitor that restarts workers on code changes
b. Relationships and interactions
1. Producer → Broker: actor.send() serializes a message, pushes to broker
2. Worker → Broker: worker pulls/consumes messages
3. Worker → Middleware: message flows through each middleware component
4. Worker → Actor: final handler executes user-defined function
5. Worker → Results Backend: on completion or failure, result state is stored
c. Architectural patterns & principles
– Modular design: broker abstraction, middleware pipeline, pluggable backends
– Separation of concerns: sending vs. executing, core vs. extensions
– Middleware chain: decorates message handling steps in a consistent order
– Back-pressure & rate limiting: configurable limits via middleware
d. Technologies & frameworks
– Python 3, setuptools for packaging
– pika and aioredis for RabbitMQ/Redis communication
– Click (or argparse) for CLI
– Lua scripts for Redis broker internals

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Services
    subgraph "External Services"
        RabbitMQ["RabbitMQ Broker"]:::external
        RedisExt["Redis Broker"]:::external
    end

    %% Application Layer
    Producer["Application / Producer"]:::external
    CLI["CLI / Launcher"]:::core
    Watcher["Watcher (File‐change Monitor)"]:::optional

    %% Core Library
    subgraph "Core Library"
        BrokerAbst["Broker Abstraction"]:::core
        Worker["Worker Process Loop"]:::core
        ActorMsg["Actor Definition & Message Model"]:::core
        Encoder["Encoding/Serialization"]:::core
        Common["Core Utilities & Compatibility"]:::core
        CanteenComp["Composition Support (Canteen)"]:::extension
    end

    %% Brokers
    subgraph "Broker Implementations"
        RabbitImp["RabbitMQ Broker Impl."]:::extension
        RedisImp["Redis Broker Impl."]:::extension
        StubBroker["Stub Broker Impl."]:::extension
    end

    %% Middleware
    subgraph "Middleware Pipeline"
        MiddlewareCore["Middleware Pipeline Core"]:::extension
        MiddlewareMods["Individual Middleware Modules"]:::extension
    end

    %% Results
    subgraph "Results Backends"
        ResultsAbst["Results Backend Abstraction"]:::extension
        ResultsImpls["Results Backend Implementations"]:::extension
        ResultsMiddleware["Results Middleware Integration"]:::extension
    end

    %% Rate Limits
    subgraph "Rate Limiting"
        RateAbst["Rate Limiter Abstraction"]:::extension
        RateBackends["Rate Limiter Backends"]:::extension
        RateAlgos["Rate Limiter Algorithms"]:::extension
    end

    %% Packaging & Docs
    Packaging["Packaging & Build Config"]:::core
    Docs["Worker Arch Diagram (Mermaid)"]:::core

    %% Data Flow
    Producer -->|"actor.send()"| BrokerAbst
    BrokerAbst -->|Publishes| RabbitImp
    BrokerAbst -->|Publishes| RedisImp
    BrokerAbst -->|Publishes| StubBroker
    CLI -->|starts| Worker
    Watcher -->|monitors & restarts| Worker
    Worker -->|pulls| BrokerAbst
    Worker -->|processes| MiddlewareCore
    MiddlewareCore -->|executes| MiddlewareMods
    MiddlewareCore -->|passes message| ActorMsg
    ActorMsg -->|executes user function| CanteenComp
    Worker -->|stores results| ResultsAbst
    ResultsAbst -->|writes to| ResultsImpls
    Worker -->|rate control| RateAbst
    RateAbst -->|uses| RateBackends
    RateAbst -->|algorithms| RateAlgos

    %% Relationships between core and impl
    BrokerAbst --> RabbitImp
    BrokerAbst --> RedisImp
    BrokerAbst --> StubBroker
    ResultsAbst --> ResultsImpls
    RateAbst --> RateBackends
    RateAbst --> RateAlgos

    %% Click Events
    click CLI "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/cli.py"
    click CLI "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/__main__.py"
    click BrokerAbst "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/broker.py"
    click RabbitImp "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/brokers/rabbitmq.py"
    click RedisImp "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/brokers/redis.py"
    click RedisImp "https://github.com/bogdanp/dramatiq/tree/master/dramatiq/brokers/redis"
    click StubBroker "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/brokers/stub.py"
    click Worker "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/worker.py"
    click Watcher "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/watcher.py"
    click MiddlewareCore "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/middleware/__init__.py"
    click MiddlewareCore "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/middleware/middleware.py"
    click MiddlewareMods "https://github.com/bogdanp/dramatiq/tree/master/dramatiq/middleware/"
    click ResultsAbst "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/results/backend.py"
    click ResultsImpls "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/results/backends/redis.py"
    click ResultsImpls "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/results/backends/memcached.py"
    click ResultsImpls "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/results/backends/stub.py"
    click ResultsMiddleware "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/results/middleware.py"
    click RateAbst "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/rate_limiter.py"
    click RateBackends "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/backends/redis.py"
    click RateBackends "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/backends/memcached.py"
    click RateBackends "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/backends/stub.py"
    click RateAlgos "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/concurrent.py"
    click RateAlgos "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/window.py"
    click RateAlgos "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/barrier.py"
    click RateAlgos "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/rate_limits/bucket.py"
    click ActorMsg "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/actor.py"
    click ActorMsg "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/message.py"
    click CanteenComp "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/canteen.py"
    click CanteenComp "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/composition.py"
    click Encoder "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/encoder.py"
    click Common "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/common.py"
    click Common "https://github.com/bogdanp/dramatiq/blob/master/dramatiq/compat.py"
    click Packaging "https://github.com/bogdanp/dramatiq/blob/master/setup.py"
    click Packaging "https://github.com/bogdanp/dramatiq/blob/master/setup.cfg"
    click Packaging "https://github.com/bogdanp/dramatiq/blob/master/MANIFEST.in"
    click Docs "https://github.com/bogdanp/dramatiq/blob/master/docs/source/_static/worker_architecture.mmd"

    %% Styles
    classDef external fill:#c8e6c9,stroke:#2e7d32,color:#2e7d32;
    classDef core fill:#bbdefb,stroke:#1565c0,color:#1565c0;
    classDef extension fill:#fff9c4,stroke:#f9a825,color:#f57f17;
    classDef optional fill:#eeeeee,stroke-dasharray: 5 5,color:#616161;
```
