How ouqiang/delay-queue works
Main Components
a) External Components:
• HTTP Server (Port 9277)
• Redis Server
• Client Applications
b) Internal Components:
• Job Handler
• Bucket Manager
• Ready Queue
• Delay Queue Scanner
Component Relationships and Data Flow
a) Input Layer:
• HTTP endpoints (/push, /pop, /delete, /finish, /get)
• To the Job Handler
b) Core Processing Layer:
• Job Handler connects to both Bucket Manager and Ready Queue
• Bucket Manager interfaces with Redis (using sorted sets)
• Delay Queue Scanner periodically checks buckets
• Ready Queue manages jobs ready for processing
Specific Implementation Details
a) Job Lifecycle:
Job Completion/Deletion
b) Data Structures:
• Redis Sorted Set (score: timestamp, member: jobID)
• Job Object Structure (topic, id, delay, ttr, body)
Key Processes to Highlight
a) Job Addition Flow:
Client → HTTP API → Job Handler → Bucket Manager → Redis
b) Job Processing Flow:
Scanner → Bucket Manager → Ready Queue → Consumer → Finish/Delete
Additional Elements
Technical details:
• Redis connection details (default: 127.0.0.1:6379, DB: 1)
• TTR (Time-To-Run) handling
• Delay calculation mechanism
• Error handling paths
Notes for Implementation Details
Annotations explaining:
• Job uniqueness requirements
• Retry mechanism
• Timeout handling
• Hold connection mechanism for pop requests (180s timeout)
Remember to emphasize the scanning mechanism that checks Redis sorted sets every second for jobs that are ready to be executed, as this is a core feature of the implementation.