# hitesh22rana/mq architecture

> Architecture diagram of the GitHub repository hitesh22rana/mq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hitesh22rana/mq
- Repository: https://github.com/hitesh22rana/mq

## Overview

##### Main Components

a) Core Message Broker Components:
- Message Queue Server (gRPC)
- Channel Manager
- Storage Layer (In-memory + WAL)
- Message Validator
- Message Generator

b) Client Components:
- Publishers
- Subscribers
- Client SDKs (Go, Python)

**Top Layer: Client Layer**
**Middle Layer: Core MQ Service Layer**
**Bottom Layer: Storage Layer**

##### Specific Component Details

- Internal components:
- Channel Manager (pkg/mq/create_channel.go)
- Publisher Service (pkg/mq/publish.go)
- Subscriber Service (pkg/mq/subscribe.go)
- Message Validator (pkg/utils/validator.go)

- In-Memory Storage (pkg/storage/memory_storage.go)
- Write-Ahead Log (WAL) component
- Data flow between storage components

- Publisher Clients
- Subscriber Clients
- Supported SDK languages (Go, Python)

##### Connection and Data Flow

- gRPC connections from clients to server
- Message flow from publishers through the broker to subscribers
- Write operations to WAL before in-memory storage
- Pull-based message retrieval from subscribers

##### Additional Elements

Include:
- Configuration components (internal/config)
- Logging system (internal/logger)
- Protocol buffer definitions (proto/mq.proto)

##### Technical Details

Show:
- gRPC port (default: 50051)
- Supported message formats (Protocol Buffers)
- Configuration parameters
- WAL directory path

##### Notes Section

Notes about:
- Scalability aspects
- Fault tolerance mechanisms
- Message durability through WAL
- Client-controlled pull intervals

##### The flexibility of multiple client SDK support

This system design should clearly communicate both the high-level architecture and the important implementation details that make this message queue broker unique, particularly its pull-based nature and WAL implementation for durability.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef client fill:#90EE90,stroke:#333,stroke-width:2px
    classDef core fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef storage fill:#FFD700,stroke:#333,stroke-width:2px
    classDef config fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef proto fill:#F08080,stroke:#333,stroke-width:2px

    %% Client Layer
    subgraph Clients
        GO_PUB["Go Publisher"]:::client
        GO_SUB["Go Subscriber"]:::client
        PY_PUB["Python Publisher"]:::client
        PY_SUB["Python Subscriber"]:::client
    end

    %% Core Service Layer
    subgraph CoreService
        GRPC["gRPC Server\n(Port: 50051)"]:::core
        CHAN_MGR["Channel Manager"]:::core
        PUB_SVC["Publisher Service"]:::core
        SUB_SVC["Subscriber Service"]:::core
        MSG_VAL["Message Validator"]:::core
        MSG_GEN["Message Generator"]:::core
    end

    %% Storage Layer
    subgraph Storage
        MEM_STORE["In-Memory Storage"]:::storage
        WAL["Write-Ahead Log"]:::storage
    end

    %% Support Systems
    subgraph Support
        CONFIG["Configuration"]:::config
        LOGGER["Logger"]:::config
        PROTO["Protocol Definitions"]:::proto
    end

    %% Relationships
    GO_PUB --> |gRPC| GRPC
    GO_SUB --> |gRPC| GRPC
    PY_PUB --> |gRPC| GRPC
    PY_SUB --> |gRPC| GRPC

    GRPC --> CHAN_MGR
    GRPC --> PUB_SVC
    GRPC --> SUB_SVC

    PUB_SVC --> MSG_VAL
    PUB_SVC --> MSG_GEN
    
    PUB_SVC --> WAL
    WAL --> MEM_STORE
    
    SUB_SVC --> MEM_STORE

    CONFIG --> GRPC
    LOGGER --> GRPC
    PROTO --> GRPC

    %% Click Events
    click GRPC "https://github.com/hitesh22rana/mq/blob/main/pkg/mq/grpc.go"
    click CHAN_MGR "https://github.com/hitesh22rana/mq/blob/main/pkg/mq/create_channel.go"
    click PUB_SVC "https://github.com/hitesh22rana/mq/blob/main/pkg/mq/publish.go"
    click SUB_SVC "https://github.com/hitesh22rana/mq/blob/main/pkg/mq/subscribe.go"
    click MSG_VAL "https://github.com/hitesh22rana/mq/blob/main/pkg/utils/validator.go"
    click MSG_GEN "https://github.com/hitesh22rana/mq/blob/main/pkg/utils/generator.go"
    click MEM_STORE "https://github.com/hitesh22rana/mq/blob/main/pkg/storage/memory_storage.go"
    click GO_PUB "https://github.com/hitesh22rana/mq/blob/main/examples/go/publisher/main.go"
    click GO_SUB "https://github.com/hitesh22rana/mq/blob/main/examples/go/subscriber/main.go"
    click PY_PUB "https://github.com/hitesh22rana/mq/blob/main/examples/python/publisher.py"
    click PY_SUB "https://github.com/hitesh22rana/mq/blob/main/examples/python/subscriber.py"
    click CONFIG "https://github.com/hitesh22rana/mq/blob/main/internal/config/config.go"
    click LOGGER "https://github.com/hitesh22rana/mq/blob/main/internal/logger/logger.go"
    click PROTO "https://github.com/hitesh22rana/mq/blob/main/proto/mq.proto"

    %% Legend
    subgraph Legend
        L1["Client Components"]:::client
        L2["Core Components"]:::core
        L3["Storage Components"]:::storage
        L4["Support Systems"]:::config
        L5["Protocol"]:::proto
    end
```
