How hitesh22rana/mq works
Main Components
a) Core Message Broker Components:
• Message Queue Server (gRPC)
• Channel Manager
• Storage Layer (In-memory + WAL)
• Message Validator
• Message Generator
b) Client Components:
• Publishers
• Subscribers
• Client SDKs (Go, Python)
Top Layer: Client Layer
Middle Layer: Core MQ Service Layer
Bottom Layer: Storage Layer
Specific Component Details
• Internal components:
• Channel Manager (pkg/mq/create_channel.go)
• Publisher Service (pkg/mq/publish.go)
• Subscriber Service (pkg/mq/subscribe.go)
• Message Validator (pkg/utils/validator.go)
• In-Memory Storage (pkg/storage/memory_storage.go)
• Write-Ahead Log (WAL) component
• Data flow between storage components
• Publisher Clients
• Subscriber Clients
• Supported SDK languages (Go, Python)
Connection and Data Flow
• gRPC connections from clients to server
• Message flow from publishers through the broker to subscribers
• Write operations to WAL before in-memory storage
• Pull-based message retrieval from subscribers
Additional Elements
Include:
• Configuration components (internal/config)
• Logging system (internal/logger)
• Protocol buffer definitions (proto/mq.proto)
Technical Details
Show:
• gRPC port (default: 50051)
• Supported message formats (Protocol Buffers)
• Configuration parameters
• WAL directory path
Notes Section
Notes about:
• Scalability aspects
• Fault tolerance mechanisms
• Message durability through WAL
• Client-controlled pull intervals
The flexibility of multiple client SDK support
This system design should clearly communicate both the high-level architecture and the important implementation details that make this message queue broker unique, particularly its pull-based nature and WAL implementation for durability.