# ouqiang/delay-queue architecture

> Architecture diagram of the GitHub repository ouqiang/delay-queue, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ouqiang/delay-queue
- Repository: https://github.com/ouqiang/delay-queue

## Overview

##### Main Components

a) External Components:
- HTTP Server (Port 9277)
- Redis Server
- Client Applications

b) Internal Components:
- Job Handler
- Bucket Manager
- Ready Queue
- Delay Queue Scanner

##### Component Relationships and Data Flow

a) Input Layer:
- HTTP endpoints (/push, /pop, /delete, /finish, /get)
- To the Job Handler

b) Core Processing Layer:
- Job Handler connects to both Bucket Manager and Ready Queue
- Bucket Manager interfaces with Redis (using sorted sets)
- Delay Queue Scanner periodically checks buckets
- Ready Queue manages jobs ready for processing

##### Specific Implementation Details

a) Job Lifecycle:
##### Job Completion/Deletion

b) Data Structures:
- Redis Sorted Set (score: timestamp, member: jobID)
- Job Object Structure (topic, id, delay, ttr, body)

##### Key Processes to Highlight

a) Job Addition Flow:
**Client → HTTP API → Job Handler → Bucket Manager → Redis**

b) Job Processing Flow:
**Scanner → Bucket Manager → Ready Queue → Consumer → Finish/Delete**

##### Additional Elements

Technical details:
- Redis connection details (default: 127.0.0.1:6379, DB: 1)
- TTR (Time-To-Run) handling
- Delay calculation mechanism
- Error handling paths

##### Notes for Implementation Details

Annotations explaining:
- Job uniqueness requirements
- Retry mechanism
- Timeout handling
- Hold connection mechanism for pop requests (180s timeout)

Remember to emphasize the scanning mechanism that checks Redis sorted sets every second for jobs that are ready to be executed, as this is a core feature of the implementation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph External
        Client["Client Applications"]
        Redis[(Redis Server)]
    end

    subgraph "HTTP Layer"
        HTTP[HTTP Server]:::server
        API1["/push"]:::api
        API2["/pop"]:::api
        API3["/delete"]:::api
        API4["/finish"]:::api
        API5["/get"]:::api
        
        HTTP --> API1
        HTTP --> API2
        HTTP --> API3
        HTTP --> API4
        HTTP --> API5
    end

    subgraph "Core Processing"
        Config[Configuration Manager]:::core
        JobHandler[Job Handler]:::core
        BucketMgr[Bucket Manager]:::core
        Scanner[Delay Queue Scanner]:::core
        ReadyQ[Ready Queue]:::core
        
        JobHandler --> BucketMgr
        JobHandler --> ReadyQ
        Scanner --> BucketMgr
        Scanner --> ReadyQ
        BucketMgr --> Redis
        ReadyQ --> Redis
    end

    Client --> HTTP
    API1 & API2 & API3 & API4 & API5 --> JobHandler

    subgraph "Job Lifecycle"
        direction TB
        Creation[Job Creation]:::lifecycle
        Storage[Bucket Storage]:::lifecycle
        Ready[Ready Queue]:::lifecycle
        Processing[Consumer Processing]:::lifecycle
        Completion[Job Completion]:::lifecycle
        
        Creation --> Storage
        Storage --> Ready
        Ready --> Processing
        Processing --> Completion
    end

    classDef api fill:#87CEEB
    classDef server fill:#4169E1
    classDef core fill:#FF6B6B
    classDef lifecycle fill:#FFD700
    classDef storage fill:#90EE90

    click HTTP "https://github.com/ouqiang/delay-queue/blob/master/routers/routers.go"
    click Config "https://github.com/ouqiang/delay-queue/blob/master/config/config.go"
    click JobHandler "https://github.com/ouqiang/delay-queue/blob/master/delayqueue/job.go"
    click BucketMgr "https://github.com/ouqiang/delay-queue/blob/master/delayqueue/bucket.go"
    click ReadyQ "https://github.com/ouqiang/delay-queue/blob/master/delayqueue/ready_queue.go"
    click Scanner "https://github.com/ouqiang/delay-queue/blob/master/delayqueue/delay_queue.go"
    click Redis "https://github.com/ouqiang/delay-queue/blob/master/delayqueue/redis.go"

    subgraph Legend
        L1[API Endpoint]:::api
        L2[Core Component]:::core
        L3[Storage]:::storage
        L4[Lifecycle State]:::lifecycle
    end
```
