# anshmehtamm/ripplemq architecture

> Architecture diagram of the GitHub repository anshmehtamm/ripplemq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anshmehtamm/ripplemq
- Repository: https://github.com/anshmehtamm/ripplemq

## Overview

- mq-broker – the broker service responsible for maintaining the messaging cluster, implementing metadata management, partition assignment, and employing RAFT for cluster and partition consensus.
- mq-common – a shared module that provides client APIs and core messaging functionalities for producers and consumers.

##### Main components
a. Broker Cluster (mq-broker module):
– Broker Service: Implemented in classes such as BrokerServer, BrokerRpcClient, and BrokerMetadataManager.
– RAFT Service for Cluster Metadata: Shown in packages under metadata/raft with PartitionRaftServer, TopicsRaftServer, and associated state machines.
– RAFT Service for Partition Management: Involves partition replication and offset management.
– Configuration: Managed through ClusterConfig and ClusterConfigManager, with configuration files (e.g., cluster_config.yaml).
b. Client API (mq-common module):
– Producer and Consumer Client Implementations (ProducerClientImpl, ConsumerClientImpl).
– Request and metadata handling classes for message append, batch read, consumer offset updates, etc.
c. Sample Producer and Consumer Applications:
– Demonstration apps (sample-producer and sample-consumer) that show how external applications can leverage mq-common to produce or consume messages.
d. Infrastructure Elements:
– Docker configuration (Dockerfile, docker-compose.yml) to set up and deploy the broker cluster.
– Build/Dependency Management: Maven (pom.xml files in each module).

##### Relationships and interactions
– Clients (Producers & Consumers) rely on the mq-common API to interface with the messaging system.
– The mq-common clients interact with broker services via network calls (likely using RPC mechanisms defined in BrokerRpcClient among others).
– Broker nodes communicate with one another using RAFT-based protocols for maintaining metadata consistency, partition assignments, and handling replication.
– The configuration and metadata management components ensure the cluster is coordinated, with a clear separation between topics metadata and individual partition management.

##### Architectural patterns and design principles
– Distributed Systems & Consensus: RAFT protocol implementation for both cluster and partition management is a key design element.
– Modular separation: Clear modular separation between the broker services (server-side) and client APIs (mq-common) simplifies integration.
– Client-Server Architecture: The producers and consumers as external clients connect to the broker over designated protocols.
– Use of Industry Standard Tools: Maven for project dependency and Docker for containerizing services.

##### Specify relevant technologies
– Java as the primary programming language.
– Docker and Docker Compose for container orchestration.
– Maven as the build and dependency management tool.
– RAFT consensus protocol for distributed consistency.

##### Elements
– Clear labels for every component, e.g., “Broker Server”, “Producer Client”, “Consumer Client”, “Cluster RAFT Module”, “Partition RAFT Module”, “Configuration Manager”.
- From sample-producer to mq-common (ProducerClientImpl) then to one or multiple BrokerServer instances.
- From BrokerServer to internal RAFT modules for metadata/partition synchronization.
- From sample-consumer to mq-common (ConsumerClientImpl) then to the broker cluster.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Applications
    subgraph "External Applications"
        SP["Sample Producer App"]:::sample
        SC["Sample Consumer App"]:::sample
    end

    %% Client API Layer (mq-common)
    subgraph "Client API (mq-common)"
        PClient["ProducerClientImpl"]:::client
        CClient["ConsumerClientImpl"]:::client
        MetaClient["MetadataClient"]:::client
    end

    %% Broker Cluster (mq-broker)
    subgraph "Broker Cluster (mq-broker)"
        %% Broker Service Components
        subgraph "Broker Service"
            BS["Broker Server"]:::broker
            BRPC["Broker RPC Client"]:::broker
            BMM["Broker Metadata Manager"]:::broker
        end

        %% RAFT Services Subgraph
        subgraph "RAFT Services"
            TSRV["Topics Raft Server"]:::raft
            TSM["Topics State Machine"]:::raft
            PRSV["Partition Raft Server"]:::raft
            PSM["Partition State Machine"]:::raft
            RProc["Request Processors"]:::raft
        end

        %% Configuration Management Subgraph
        subgraph "Configuration Management"
            CConfig["Cluster Config"]:::config
            CCM["Cluster Config Manager"]:::config
            YAML["cluster_config.yaml"]:::config
        end
    end

    %% Infrastructure Elements
    subgraph "Infrastructure"
        DockerFile["Dockerfile"]:::infra
        DockerCompose["docker-compose.yml"]:::infra
        Maven["Maven Build"]:::infra
    end

    %% Data Flow Connections
    SP -->|"calls"| PClient
    SC -->|"calls"| CClient
    PClient -->|"invokes"| BS
    CClient -->|"invokes"| BS
    MetaClient -->|"supports"| BS

    BS -->|"uses"| BRPC
    BS -->|"manages"| BMM
    BS -->|"loads"| CConfig

    BMM -->|"syncs with"| TSRV
    BMM -->|"syncs with"| PRSV
    TSRV -->|"operates with"| TSM
    PRSV -->|"operates with"| PSM
    BS -->|"processes via"| RProc

    CConfig -->|"initializes"| CCM
    CCM -->|"reads"| YAML

    %% Infrastructure associations
    DockerCompose -->|"deploys"| BS
    DockerFile -->|"builds"| BS
    Maven -->|"builds"| BS
    Maven -->|"builds"| PClient
    Maven -->|"builds"| CClient
    Maven -->|"builds"| SP
    Maven -->|"builds"| SC

    %% Styling Classes
    classDef external fill:#fce,stroke:#333,stroke-width:2px;
    classDef client fill:#cff,stroke:#333,stroke-width:2px;
    classDef broker fill:#cfc,stroke:#333,stroke-width:2px;
    classDef raft fill:#ccf,stroke:#333,stroke-width:2px;
    classDef config fill:#fcf,stroke:#333,stroke-width:2px;
    classDef infra fill:#ffc,stroke:#333,stroke-width:2px;
    classDef sample fill:#fcc,stroke:#333,stroke-width:2px;

    %% Click Events for Broker Service
    click BS "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/broker/BrokerServer.java"
    click BRPC "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/broker/BrokerRpcClient.java"
    click BMM "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/broker/BrokerMetadataManager.java"

    %% Click Events for RAFT Services
    click PRSV "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/metadata/raft/PartitionRaftServer.java"
    click TSRV "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/metadata/raft/TopicsRaftServer.java"
    click PSM "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/metadata/raft/PartitionStateMachine.java"
    click TSM "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/metadata/raft/TopicsStateMachine.java"
    click RProc "https://github.com/anshmehtamm/ripplemq/tree/main/mq-broker/src/main/java/metadata/raft/request/processor"

    %% Click Events for Configuration Management
    click YAML "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/config/cluster_config.yaml"
    click CConfig "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/config/ClusterConfig.java"
    click CCM "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/src/main/java/config/ClusterConfigManager.java"

    %% Click Events for Client API
    click PClient "https://github.com/anshmehtamm/ripplemq/blob/main/mq-common/src/main/java/client/ProducerClientImpl.java"
    click CClient "https://github.com/anshmehtamm/ripplemq/blob/main/mq-common/src/main/java/client/ConsumerClientImpl.java"
    click MetaClient "https://github.com/anshmehtamm/ripplemq/blob/main/mq-common/src/main/java/metadata/MetadataClient.java"

    %% Click Events for Sample Applications
    click SP "https://github.com/anshmehtamm/ripplemq/blob/main/sample-producer/src/main/java/org/example/Main.java"
    click SC "https://github.com/anshmehtamm/ripplemq/blob/main/sample-consumer/src/main/java/org/example/Main.java"

    %% Click Events for Infrastructure
    click DockerFile "https://github.com/anshmehtamm/ripplemq/tree/main/mq-broker/Dockerfile"
    click DockerCompose "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/docker-compose.yml"
    click Maven "https://github.com/anshmehtamm/ripplemq/blob/main/mq-broker/pom.xml"
```
