# hibiken/asynq architecture

> Architecture diagram of the GitHub repository hibiken/asynq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hibiken/asynq
- Repository: https://github.com/hibiken/asynq

## Overview

##### Project Type and Purpose
- Asynq is a distributed asynchronous task processing library in Go. Its purpose is to allow clients to enqueue tasks into Redis and to have dedicated worker servers pull these tasks for concurrent processing.
- The README describes use cases like scheduling, retrying failed tasks, handling periodic tasks, and even aggregating small tasks. It also mentions integrations (e.g., Redis, Prometheus, Web UI, CLI) indicating that the library is built for robust distributed systems.

##### Analysis of File Structure
- At the top level, you find key Go source files (aggregator.go, asynq.go, client.go, server.go, etc.) that indicate the core components of the task queue library.
- The “internal” directory contains sub-packages that support base functionality, context management, logging, error handling, proto definitions for task serialization, and Redis database access (rdb). These implement the lower-level engine parts.
- There are also tool directories (tools/asynq, tools/metrics_exporter) which implement a command line interface and a dashboard for monitoring and inspecting tasks.
- The presence of test files for individual components (e.g., aggregator_test.go, processor_test.go, etc.) shows careful testing integration.
- Files like Makefile, go.mod, and GitHub workflows indicate that the project follows a standardized build and CI/CD process.

##### Key System Components & Relationships
a. Client Component:
– Responsible for creating tasks with types and payloads.
– Interacts with Redis directly by enqueuing tasks.

b. Server/Worker Component:
– Runs as a worker pool that pulls tasks from Redis queues.
– Each worker starts a goroutine for individual tasks and processes them concurrently.
– The server is configurable (e.g., concurrency settings, queue priorities).

c. Redis Backend:
– Serves as the message broker and persistent storage for queued tasks.
– It is central to communication between the client and the server.

d. Additional Components:
– “Scheduler” and “Periodic Task Manager” components that support delayed and scheduled tasks.
– “Healthcheck”, “Inspector”, “Recoverer”, and “Heartbeat” modules to ensure robustness and system monitoring.
– “CLI and Web UI Tools” that provide interfaces for monitoring tasks (e.g., Asynqmon) and administrating queues. Show these as external monitoring/administration tools interfacing with Redis and the worker server.

##### Architectural Patterns and Design Principles
- The library implements an asynchronous, decoupled producer-consumer pattern where the client enqueues tasks and the server processes them independently.
- It uses middleware-style handler registration (like ServeMux) to separate various task types and processing rules.
- There is an emphasis on high availability (e.g., Redis sentinel support) and horizontal scaling for workers.
- The design ensures at-least-once execution with retries and error handling strategies.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Task Producers
    subgraph "Task Producers"
        CLIENT["Client (Task Enqueuer)"]:::producer
    end

    %% Data Broker
    subgraph "Data Broker"
        REDIS["Redis (Message Broker)"]:::database
    end

    %% Worker Server and Internal Processing
    subgraph "Worker Server"
        WORKER["Worker Server"]:::worker
        PROCESSOR["Handler/Processor"]:::workerSub
        RECOVERER["Recoverer"]:::workerSub
        SCHEDULER["Scheduler"]:::workerSub
        PERIODIC["Periodic Task Manager"]:::workerSub
        HEALTH["Healthcheck"]:::monitor
        HEARTBEAT["Heartbeat"]:::monitor
        INSPECTOR["Inspector"]:::monitor
    end

    %% Administration and Monitoring Tools
    subgraph "Admin & Monitoring Tools"
        CLI["CLI & Web UI Tools"]:::admin
    end

    %% Data Flows
    CLIENT -->|"EnqueueTask"| REDIS
    REDIS -->|"DistributeTask"| WORKER
    WORKER -->|"ProcessTask"| PROCESSOR
    WORKER -->|"HandleFailures"| RECOVERER
    PROCESSOR -->|"ScheduleTask"| SCHEDULER
    PROCESSOR -->|"SchedulePeriodic"| PERIODIC
    WORKER -->|"Monitor"| HEALTH
    WORKER -->|"HeartbeatCheck"| HEARTBEAT
    WORKER -->|"Inspection"| INSPECTOR
    CLI -->|"AdministerMonitor"| REDIS
    CLI -->|"AdministerMonitor"| WORKER

    %% Click Events
    click CLIENT "https://github.com/hibiken/asynq/blob/master/client.go"
    click WORKER "https://github.com/hibiken/asynq/blob/master/server.go"
    click PROCESSOR "https://github.com/hibiken/asynq/blob/master/processor.go"
    click RECOVERER "https://github.com/hibiken/asynq/blob/master/recoverer.go"
    click REDIS "https://github.com/hibiken/asynq/tree/master/internal/rdb"
    click SCHEDULER "https://github.com/hibiken/asynq/blob/master/scheduler.go"
    click PERIODIC "https://github.com/hibiken/asynq/blob/master/periodic_task_manager.go"
    click HEALTH "https://github.com/hibiken/asynq/blob/master/healthcheck.go"
    click HEARTBEAT "https://github.com/hibiken/asynq/blob/master/heartbeat.go"
    click INSPECTOR "https://github.com/hibiken/asynq/blob/master/inspector.go"
    click CLI "https://github.com/hibiken/asynq/tree/master/tools/asynq"

    %% Styles
    classDef producer fill:#85C1E9,stroke:#1F618D,stroke-width:2px;
    classDef database fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef worker fill:#ABEBC6,stroke:#196F3D,stroke-width:2px;
    classDef workerSub fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef monitor fill:#FAD7A0,stroke:#CA6F1E,stroke-width:2px;
    classDef admin fill:#F5B7B1,stroke:#943126,stroke-width:2px;
```
